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From  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 


The  Original  Better  Buying 
David  Packard  Acquisition 

Frank  Kendall 


Power- 
Rules  1971 


n  this  article,  I  thought  I  would  give  us  all 
a  break  from  our  budget  woes,  sequestra¬ 
tion,  and  continuing  resolutions— issues  I 
hope  will  be  resolved  before  this  goes  to 
print. 

In  1971, 1  graduated  from  West  Point.  This  was  also  the  same 
year  that  David  Packard,  the  Packard  in  Hewlett  Packard, 
who  was  then  the  Deputy  Secretary  of  Defense  (there  was 
no  Under  Secretary  for  Acquisition),  published  his  rules  for 
Defense  Acquisition.  I  wouldn't  say  there  has  been  nothing 
new  under  the  sun  since  then,  but  some  things  do  endure. 

Recall  that  by  1971  we  had  already  been  to  the  moon,  and  the 
digital  age,  enabled  by  solid  state  electronics,  had  just  begun. 
By  the  fall  of  1971,  I  was  at  Caltech  where  I  designed  logic 
circuits  using  solid  state  integrated  components  that  included 
a  few  specific  logic  functions— several  orders  of  magnitude 
from  current  technology,  and  I  was  reducing  experimental 
data  using  the  first  engineering  math  function  digital  calcu¬ 
lator.  My  slide  rule  had  become  obsolete.  Deputy  Secretary 
Packard's  rules,  however,  still  resonate.  I  recently  had  them 
put  on  a  poster  and  hung  it  in  the  Pentagon  in  the  room  we 
use  for  Defense  Acquisition  Board  (DAB)  meetings.  Here  they 
are  with  a  little  commentary  from  both  David  Packard  and 
me.  You  should  recognize  a  number  of  areas  of  overlap  with 
Better  Buying  Power. 

1.  Help  the  Services  Do  a  Better  Job. 

Improvement  in  the  development  and  acquisition  of  new  weapons 
systems  will  be  achieved  to  the  extent  the  Services  are  willing  and 
able  to  improve  their  management  practices.  The  Services  have  the 
primary  responsibility  to  get  the  job  done.  OSD  offices  should  see 
that  appropriate  policies  are  established  and  evaluate  the  perfor¬ 
mance  of  the  Services  in  implementing  these  policies. 

I  continue  to  struggle  with  achieving  the  appropriate  degree 
of  staff  "oversight,"  but  I  certainly  agree  with  this  sentiment. 
Services  manage  programs.  As  Defense  Acquisition  Executive 
(DAE),  I  set  policy  and  I  make  specific  decisions  about  major 
investment  commitments  for  large  programs,  usually  at  Mile¬ 
stone  Reviews.  The  staff  supports  me  in  those  decisions,  and 
I  expect  solid  independent  "due  diligence"  assessments  for 


Defense  AT&L:  May-June  2013 


2 


those  decisions  from  the  staff  of  the  Office  of  the  Secretary  of 
Defense  (OSD).  All  other  staff  activities  should  be  about  help¬ 
ing  the  Services  be  more  effective,  ensuring  that  our  policies 
are  well  defined,  and  getting  feedback  on  what  works  and  what 
needs  to  be  improved  in  our  acquisition  practices. 

2.  Have  Good  Program  Managers  with 
Authority  and  Responsibility. 

If  the  Services  are  to  do  a  better  job,  they  must  assign  better  pro¬ 
gram  managers  to  these  projects.  These  managers  must  be  given 
an  appropriate  staff  and  the  responsibility  and  the  authority  to 
do  the  job,  and  they  must  be  kept  in  the  job  long  enough  to  get 
something  done. 

I  don't  know  anything  more  basic  and  important  to  our  suc¬ 
cess  than  this  imperative.  Having  seen  more  than  4  decades 
of  defense  acquisition  policy  changes,  I  am  absolutely  con¬ 
vinced  that  nothing  matters  as  much  as  competent,  profes¬ 
sional  leadership.  Once  you  have  that,  the  rest  is  details.  It 


purpose.  I've  seen  several  variations  of  this;  during  my  first  tour 
of  duty  in  OSD,  we  used  "Cost  as  an  Independent  Variable"  to 
try  to  capture  this  idea.  The  approach  we  are  using  now  relies 
on  the  affordability  caps  (which  are  based  on  future  budget 
expectations— not  on  cost  estimates)  that  we  are  establishing 
early  in  the  design  process  or  product  life  cycle  (Milestones 
A  and  B).  The  requirement  to  deliver  products  that  meet  the 
affordability  caps  is  intended  to  force  requirements  prioritiza¬ 
tion  and  trade-offs  among  competing  needs.  I  plan  to  insert  a 
Requirements  Decision  Point  prior  to  Milestone  (MS)  B  to  help 
facilitate  this.  I  will  continue  to  put  these  affordability  caps  in 
place  and  will  be  enforcing  them  over  the  next  several  years. 
For  non-ACAT  I  programs,  the  Services  and  Agencies  should 
be  doing  the  same. 

4.  Make  the  First  Decision  Right. 

The  initial  decision  to  go  ahead  with  full-scale  development  of  a 
particular  program  is  the  most  important  decision  of  the  program. 
If  this  decision  is  wrong,  the  program  is  doomed  to  failure.  To  make 


was  my  concern  for  the  professionalism  of  the  acquisition 
workforce  that  led  to  the  inclusion  of  an  additional  category 
of  initiatives  focused  on  our  workforce  in  BBP  2.0.  We  have 
a  lot  of  good,  even  great,  extremely  dedicated,  professionals 
working  in  Defense  Acquisition.  But  we  need  a  deeper  bench, 
and  every  one  of  us  can  improve  on  our  own  abilities.  In  the 
tough  budget  climate  of  today,  managers  at  all  levels,  includ¬ 
ing  Military  Department  and  Agency  leadership,  should  pay 
a  great  deal  of  attention  to  retaining  and  managing  our  talent 
pool.  At  the  tactical  level.  I'm  looking  for  some  opportunities  to 
take  a  "skunk  works"-like  approach  to  a  pilot  program  in  each 
Service.  The  key  to  implementing  this  approach,  however,  and 
what  I  want  to  be  sure  of  before  I  authorize  it,  will  be  a  highly 
qualified  and  appropriately  staffed  government  team  that  will 
be  with  the  project  until  the  product  is  delivered. 

3.  Control  Cost  by  Trade-Offs. 

The  most  effective  way  to  control  the  cost  of  a  development  pro¬ 
gram  is  to  make  practical  trade-offs  between  operating  require¬ 
ments  and  engineering  design. 

The  affordability  as  a  requirement  element  of  Better  Buying 
Power  Is  intended  to  provide  a  forcing  function  for  just  this 


this  decision  correctly  generally  will  require  that  the  program  be 
kept  in  advanced  development  long  enough  to  resolve  the  key 
technical  uncertainties,  and  to  see  that  they  are  matched  with  key 
operating  requirements  before  the  decision  to  go  ahead  is  made. 

I  have  long  regarded  the  decision  to  enter  Engineering  and 
Manufacturing  Development  (EMD)  as  the  single  most  impor¬ 
tant  decision  in  a  program's  life  cycle.  The  name  has  changed 
several  times  over  my  career,  and  Deputy  Secretary  Packard 
refers  to  it  as  full-scale  development— but  we  are  talking  about 
the  commitment  to  go  on  contract  for  design  of  a  producible 
product  that  meets  stated  requirements,  engineering  develop¬ 
ment  test  articles,  and  for  the  tests  that  will  be  necessary  to 
confirm  performance  prior  to  starting  production. 

At  this  point,  we  are  committing  to  on  average  about  10  per¬ 
cent  to  20  percent  of  the  product's  life-cycle  cost  to  years  of 
development  work,  and  to  getting  a  product  that  we  will  field 
ready  for  production.  Among  the  most  disturbing  sources  of 
waste  in  our  system  are  the  programs  we  put  into  EMD,  spend 
billions  on,  and  then  cancel— sometimes  before  EMD  is  com¬ 
plete  and  sometimes  after  some  initial  production.  Part  of  get¬ 
ting  this  decision  right  (in  addition  to  affordability)  is  having  the 
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risk  associated  with  the  product  and  its  requirements  under 
control  and  sufficiently  understood  and  reduced  so  EMD  can 
be  executed  efficiently  and  successfully.  In  recent  years,  we 
have  focused  on  the  Technology  Readiness  Level  (TRL)  as  a 
metric  for  maturity.  I  find  this  metric  to  be  useful,  but  not  ad¬ 
equate  to  the  task  of  assuring  readiness  to  enter  EMD,  and  not 
a  substitute  for  a  thorough  understanding  of  the  actual  risk  in 
the  program— necessary  but  not  sufficient,  in  other  words.  In 
addition  to  technology  risk,  we  have  to  manage  engineering 
and  integration  risks.  More  importantly,  we  have  to  deeply 
understand  the  actual  risk,  what  It  implies,  and  what  the  tools 
are  to  mitigate  it  before  and  during  EMD.  I  commissioned  a  re¬ 
view  of  programs  transitioning  from  Technology  Development 
into  EMD  over  a  year  ago  and  discovered  we  are  not  paying 
adequate  attention  to  the  actual  risk  associated  with  the  actual 
product  we  intend  to  acquire.  In  many  cases,  industry  was 
not  being  incentivized  to  reduce  the  actual  risk  in  a  product  it 


The  decision  to  enter  production  at  MS  C  is  different.  Here 
the  emphasis  is  on  whether  the  design  meets  requirements 
and  Is  stable.  I  would  regard  this  decision  as  a  close  second  to 
the  EMD  decision  in  importance.  Once  we  start  production, 
we  are  effectively  committed,  and  it  will  be  very  difficult  to 
stop.  I  seriously  considered  stopping  F-35  production  a  year 
ago,  but  I  believe  I  made  the  right  decision  to  continue.  We 
shouldn't  put  ourselves  in  the  position  of  having  to  make  that 
sort  of  a  choice. 

Before  the  commitment  to  production,  the  ability  to  meet  re¬ 
quirements  and  the  stability  of  the  design  should  be  demon¬ 
strated  by  developmental  testing  of  EMD  prototypes  that  are 
close  to  the  production  design.  Some  degree  of  concurrency 
usually  Is  acceptable;  all  testing  doesn't  usually  have  to  be 
complete  before  the  start  of  low-rate  production.  The  degree 
of  concurrency  will  vary  with  the  urgency  of  the  need  for  the 


In  many  cases,  industry  was  not  being  incentivized  to  reduce  the 
actual  Hsk  in  a  product  it  would  produce;  it  was  being  incentivized 
to  claim  a  TRL  and  to  do  a  demonstration. 


would  produce;  it  was  being  incentivized  to  claim  a  TRL  and 
to  do  a  demonstration.  This  Isn't  necessarily  the  same  thing 
as  reducing  the  risk  in  an  actual  product.  The  label  of  a  TRL 
isn't  enough  to  ensure  that  the  risks  of  a  product  development 
are  under  control;  we  have  to  look  deeper.  This  decision  Is  too 
important  to  get  wrong. 

5.  Fly  Before  You  Buy. 

Engineering  development  must  be  completed  before  substantial 
commitment  to  production  is  made. 

If  you  have  read  any  article  about  the  F-35  Joint  Strike  Fighter 
in  the  last  year,  you  probably  saw  a  quote  of  my  comment 
about  "acquisition  malpractice."  I  was  talking  specifically  about 
the  decision  to  enter  production  well  before  the  first  flight 
of  a  production  representative  EMD  prototype.  The  earlier 
Milestones  in  our  Materiel  Development  Decisions  (MDD) 
system  for  weapons  acquisition— MS  A  and  MS  B— generally 
are  based  on  planning  documents  and  analysis.  MS  B  also  Is 
based  on  risk-reduction  activities,  but  if  these  have  been  com¬ 
pleted,  the  balance  of  the  review  Is  about  intended  business 
approaches,  engineering,  test  planning,  and  funding  adequacy. 


product  and  the  specific  risks  remaining.  But  as  a  general  prac¬ 
tice,  we  should  "fly  before  we  buy." 

6.  Put  More  Emphasis  on  Hardware,  Less  on 
Paper  Studies. 

Logistics  support,  training,  and  maintenance  problems  must  be 
considered  early  in  the  development,  but  premature  implementa¬ 
tion  of  these  matters  tends  to  be  wasteful. 

Most  of  the  costs  of  our  products  are  neither  development  nor 
production  costs.  It  Is  support  costs  that  predominate.  These 
costs  do  need  to  be  considered  up  front,  early  in  the  require¬ 
ments  and  design  processes  and  as  the  acquisition  strategy 
Is  being  formulated.  They  drive  considerations  of  the  data  and 
property  rights  we  will  acquire  and  the  implementation  of  open 
systems  and  modular  designs  (all  features  of  Better  Buying 
Power).  While  we  should  avoid  setting  up  support  functions 
too  much  in  advance  of  need,  we  also  should  ensure  that  the 
ability  to  meet  support  requirements  is  designed  in  and  tested 
at  the  appropriate  places  in  the  development  program,  and  we 
must  ensure  that  an  adequate  budget  will  be  available  to  sus¬ 
tain  the  product.  Better  Buying  Power's  affordability  caps  on 
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sustainment  costs  are  designed  to  ensure  that  these  upfront 
analyses  are  conducted  early  in  development,  preferably  while 
there  is  still  competition  for  the  development  work,  and  before 
the  design  concept  has  matured  to  the  point  that  trade-offs  to 
improve  supportability  no  longer  are  possible. 

7.  Eliminate  Total  Package  Procurement. 

It  is  not  possible  to  determine  the  production  cost  of  a  complex 
new  weapon  before  it  is  developed.  The  total  package  procurement 
procedure  is  unworkable.  It  should  not  be  used. 

Total  Package  Procurement  is  one  of  those  acquisition  ideas 
that  come  along  occasionally  and  are  embraced  for  a  time  until 
it  becomes  apparent  they  are  not  panaceas.  I'm  speculating, 
but  I  would  guess  the  Deputy  Secretary  had  seen  some  disas¬ 
ters  come  out  of  this  approach.  The  idea  is  to  get  prices  (as  op¬ 
tions,  presumably)  for  the  production  run  at  the  time  we  start 
development.  I'm  not  quite  as  pessimistic  as  Deputy  Secretary 
Packard  was  about  the  ability  to  predict  production  costs,  but 
I'm  pretty  close.  We  are  tempted  occasionally  to  ask  for  pro¬ 
duction  prices  as  options  at  the  time  we  are  doing  a  competi¬ 
tive  down-select  for  EMD.  This  is  tempting  because  we  can 
take  advantage  of  competitive  pressure  that  we  will  lose  after 
we  enter  EMD.  While  I  wouldn't  close  out  this  idea  entirely  as 
Deputy  Secretary  Packard  did  in  this  rule,  I  think  we  have  to 
consider  this  approach  carefully  before  adopting  it.  There  are 
other  ways  to  provide  incentives  to  control  production  costs, 
and  we  need  to  consider  the  full  range  of  options  and  the  pros 
and  cons  and  the  risks  associated  with  them  before  we  decide 
on  an  acquisition  strategy  or  a  contract  structure  for  a  specific 
product.  BBP  2.0  takes  this  approach. 

8.  Use  the  Type  of  Contract  Appropriate  for 
the  Job. 

Development  contracts  for  new  major  weapons  systems  should  be 
cost-incentive  type  contracts,  (a)  Cost  control  of  a  development 
program  can  be  achieved  by  better  management,  (b)  A  prime  ob¬ 
jective  of  every  development  program  must  be  to  minimize  the 
life-cycle  cost  as  well  as  the  production  cost  of  the  article  or  system 
being  developed,  (c)  Price  competition  is  virtually  meaningless  in 
selecting  a  contractor  for  a  cost-incentive  program.  Other  factors 
must  control  the  selection. 

We  seem  to  work  in  20-year  cycles.  In  1971,  David  Packard 
supported  the  use  of  cost-plus  contracts  for  development. 
About  20  years  later  in  the  late  1980s,  we  tried  a  policy  or 
requiring  firm  fixed-price  contracts  for  development.  I  lived 
that  dream  from  the  perspective  of  having,  in  the  early  1990s, 
to  extricate  the  Department  from  the  disasters  that  ensued— 
not  least  among  them  the  Navy's  A-12  program  cancelation, 
which  still  is  in  litigation  more  than  20  years  later.  Fast  forward 
another  20  years,  and  we  are  seeing  suggestions  of  using  this 
approach  again.  Recently,  I  wrote  at  length  about  the  times 


With  the  assistance  of  the  Office  of  the  Secretary  of  De¬ 
fense,  Defense  AT&L  publishes  the  names  of  incoming  and 
outgoing  program  managers  for  major  defense  acquisition 
programs  (MDAPs)  and  major  automated  information 
system  (MAIS)  programs.  This  announcement  lists  all 
such  changes  of  leadership,  for  both  civilian  and  military 
program  managers  for  January  and  February  2013,  with 
some  dating  to  December  of  last  year. 

Marine  Corps 

Col.  Steven  Girard  relieved  Col.  Harry  Hewson  as  pro¬ 
gram  manager  for  USMC  Light/Attack  Helicopter  Pro¬ 
gram  (PMA  276)  on  Feb.  1. 

Air  Force 

Lt.  Col.  Michael  W.  Bishop  relieved  Scott  C.  Hardimann 

as  program  manager  of  the  Global  Broadcast  Service  on 
Feb.  11. 

Col.  Shaun  Q.  Morris  assumed  the  duty  of  program  man¬ 
ager  for  the  KC-46  Tanker  as  part  of  the  Air  Force  Materiel 
Command  reorganization  on  Jan.  14. 

Lt.  Gen.  Christopher  C.  Bogdan  relieved  Vice  Adm.  David 
J.  Venlet  as  program  manager  for  the  F-35  Lightning  II  on 
Dec.  6,  2012. 

Mr.  Randall  Culpepper  assumed  the  duty  of  program  ex¬ 
ecutive  officer  of  combat  and  mission  support  on  Dec.  2, 
2012.^ 


when  a  fixed-price  development  approach  might  be  appro¬ 
priate,  and  I  won't  repeat  that  material  here.  There  are  times 
when  fixed  price  is  the  right  approach  to  development  con¬ 
tracts,  but  it  Is  the  exception  rather  than  the  rule.  I  completely 
agree  with  David  Packard  that  costs  can  be  controlled  on  a 
cost-plus  contract  by  better  management.  It  requires  hands- 
on  management  and  a  willingness  to  confront  industry  about 
excessive  and  unnecessary  costs  or  activities.  It  also  requires 
strong  incentives  to  reward  the  performance  we  should  ex¬ 
pect,  coupled  with  the  will  and  expertise  to  use  those  incen¬ 
tives  effectively.  The  importance  of  controlling  life-cycle  costs 
has  been  discussed  earlier.  I  don't  entirely  agree  that  price 
competition  is  meaningless  in  selecting  a  contractor  for  a  de¬ 
velopment  contract,  but  I  do  agree  that  other  factors  should 
usually  be  of  greater  significance  to  the  government.  Most  of 
all,  I  fully  concur  with  Deputy  Secretary  Packard's  overarching 
point:  Use  the  contract  type  appropriate  for  the  job. 

If  you  get  a  chance  to  attend  a  DAB  or  DAES  meeting,  or 
just  to  come  into  the  Pentagon,  you  can  see  David  Packard's 
rules  on  the  wall  in  Room  3B912.  They  still  resonate.  We 
have  tough  jobs,  and  the  professionalism  needed  to  do  them 
effectively  is  a  constant.  There  are  no  rules  that  can  be  a 
substitute  for  that.  ^ 
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At  the  Defense  Acquisition  University,  we  spend  a  lot  of  time  with  incoming  program 
managers  (PMs)  as  they  attend  their  courses,  and  help  them  plan  strategies  for  achiev¬ 
ing  their  acquisition  goals.  What's  not  as  well  known  is  that  we  also  spend  a  lot  of  time 
in  the  workplace  with  PMs  and  their  program  teams,  collaborating  with  them  to  solve 
issues  and  to  capitalize  on  opportunities.  Based  on  that  experience,  we  would  like  to 
share  some  of  the  insights  we've  gained  from  these  collaborations.  We'll  start  with  a  short  laydown 
of  one  of  our  core  program  assist  tools,  the  Acquisition  Program  Transition  Workshop  (APTW), 
and  follow  that  with  insights  gained  from  APTWs  and  other  interactions. 

So  why  do  we  need  a  transition  workshop?  It  started  with  a  joint  Raytheon  and  DAU  effort  aimed  at  addressing  how 
to  tailor  a  new  contract  startup  for  an  increased  probability  of  success.  How  well  a  contract  startup  is  conducted 


Higbee  is  executive  director  for  Mission  Assistance  at  the  Defense  Acquisition  University.  Stewart  is  director  of  the  Major  Defense  Acquisi¬ 
tion  Program  (MDAP)  at  the  Defense  Systems  Management  College. 
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is  an  accurate  indicator  of  how  well  the  contract  will 
execute.  The  DAU-Raytheon  team  took  a  proven  Ray¬ 
theon  program  startup  process,  and  then  adapted  it  for 
general  DoD  use.  After  successfully  piloting  the  APTW 
process,  the  APTW  team  was  joined  in  a  fine-tuning  pro¬ 
cess  by  Lockheed  Martin,  Boeing,  Harris,  and  Northrop 
Grumman.  Ultimately,  Dr.  Ashton  Carter— then  under 
secretary  of  defense  for  acquisition,  technology  and  lo¬ 
gistics-signed  a  Directive  Type  Memorandum  (April  1, 
2011)  strongly  recommending  APTWs  for  new  program 
and  contract  starts,  as  well  as  for  managing  change  in 
programs  and  their  contracts  as  they  proceed  through 
the  acquisition  life  cycle. 


and  the  DAU  APTW  team  to  shape  APTW  content 
in  support  of  the  PM's  goals/expectations,  which  also 
include  the  integrated  expectations  of  his  or  her  key 
leaders. 

Lesson  Learned  No.  1:  The  PM  must  be  seen  by  the 
government-industry  APTW  participants  as  having  clear 
goal  definition,  fully  committed  to  the  APTW  process, 
and  must  actively  participate. 

Lesson  Learned  No.  2:  Listen  to  your  team.  Part  of  the 
thorough  preparation  must  be  consulting  with  the  team 
members  to  obtain  their  inputs. 


APTWs  start  with  in-depth  preparation,  including  in-  This  collaborative  preparation  establishes  actionable 
terviews  with  the  industry-government  team,  team  APTW  outcomes  honed  by  the  participants,  both 
surveys,  and  assessment  tools  covering  pertinent  is-  government  personnel  and  (where  participating) 
sues.  The  surveys,  interviews,  and  tools  allow  the  PM  contractors. 
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As  we  move  into  the  actual  workshop,  the  government  should 
enter  with  clear  expectations  about  contract  execution  for  the 
short  (next  6  months)  term,  and  a  real  understanding  of  the  inte¬ 
grated  master  plan  and  schedule.  Normally,  a  key  APTW  activity 
involves  industry  and  government  collaboration  on  preparing  a 
plan  for  a  successful  Integrated  Baseline  Review  (IBR),  with  a 
parallel  in-depth  look  at  the  integrated  master  schedule  and  the 
critical  path.  In  support  of  this  goal,  the  APTW  stresses  aligning 
both  portions  of  the  team  to  a  well-defined  RAM  (Responsibility 
Assignment  Matrix).  Planned  and  executed  well,  these  activi¬ 
ties  are  truly  a  program  jumpstart  toward  a  successful  IBR  and 
successful  contract  execution. 

So  what  do  we  see  in  our  dialog  with  DoD  program  offices? 
Probably  the  same  things  you're  seeing— lots  of  capable  pro¬ 
fessionals,  a  fairly  high  turnover  of  team  personnel.  Experi¬ 
ence  is  usually  high  at  the  senior  levels,  not  necessarily  so 
for  the  new  or  junior  persons  in  the  program.  Teams  often 
aren't  aligned— either  within  the  government  or  the  combined 
government-contractor  team. 

Change  in  government  acquisition  also  Is  a  constant  (pun  in¬ 
tended).  Change  includes  new  policies,  new  contracts,  coping 
with  resources  and  schedule  changes,  downsizing,  and  (now) 
sequestration.  What  are  some  of  the  "golden  nuggets"  we've 
taken  away  from  our  work? 

Let's  take  an  acquisition  transition  workshop  that  addresses 
the  need  to  create  a  major  new  contract: 

Lesson  Learned  No.  3:  Engaging  the  government  and  in¬ 
dustry  teams  Is  better  done  earlier  in  the  contracting  cycle 
(e.g.,  via  Broad  Area  Announcements  or  Requests  for 
Information). 

If  the  APTW  happens  later,  as  the  Request  for  Proposal  is 
drafted: 

Lesson  Learned  No.  4:  Align  your  program  team  to  optimally 
support  the  new  contract  (e.g.,  using  the  product  work  break¬ 
down  structure). 

Tell  the  potential  bidders  about  your  projected  orga¬ 
nization.  That  helps  the  contractor  better  support  the 


government  team.  It  also  supports  rapid  program  IPT  setup 
and  chartering. 

If  these  areas  aren't  covered  early,  we  repeatedly  see  a  great 
deal  more  time  and  energy  required  in  the  post-award  effort 
to  fully  align  the  government  and  contractor. 

Managing  this  part  of  the  change  equation  is  where  DAU's 
workshop  products  can  be  of  great  help.  Products  such  as 
joint  government-contractor  team  charters  seem  simple, 
but  defining  "who  is  to  do  what,  and  by  when"  is  essential 
to  deconflicting  government/industry  efforts.  It  is  also  a  key 
to  gaining  a  common  understanding  of  the  complete  body  of 
contract  work. 

Roles  and  responsibilities  aren't  always  clear  in  either  legacy 
or  new  organizations. 

Lesson  Learned  No.  5:  Focus  on  the  task  at  hand— set¬ 
ting  the  basic  organizational  structure  and  tying  together 
responsibilities. 

Just  because  things  worked  under  the  old  contract,  doesn't 
mean  they  will  continue  to  do  so  under  the  new  or  modified 
contract.  What  the  government  portion  of  the  team  expects 
to  do  needs  to  be  bounced  against  the  contractor's  con¬ 
cept  of  what  his  or  her  team  is  contracted  to  do.  Contrac¬ 
tor  roles  and  responsibilities  may  have  changed  (e.g.,  shifting 
from  development  to  production  of  a  product).  One  tool  we 
have  found  very  useful  in  working  through  these  changes  Is 
RASCI— "responsible,  accountable,  supporting,  consulted,  and 
informed."  RASCI  allows  us  to  help  you  match  your  team  to 
your  contractor  equivalents,  the  IPT  duties,  the  communica¬ 
tions  plan,  and  your  metrics . . .  helping  you  get  your  extended 
team  organized  optimally  prior  to  the  start  of  the  effort. 

Communications  Issues  seem  to  come  up  again  and  again. 

Lesson  Learned  No.  6:  Lack  of  clear  communications  always 
needs  to  be  rapidly  analyzed  and  corrected. 

Team  members  with  whom  we  talk  keep  bringing  commu¬ 
nications  to  the  forefront.  It's  about  their  perceived  lack  of 
office  or  program  communications,  poor  meeting  execution. 


Defense  AT&L:  May-June  2013 


meeting  overload,  and  disregard  of  team  analysis  and  recom¬ 
mendations,  often  resulting  in  "unmade"  decisions  or  "per¬ 
petual  decision  revisiting."  Poor-communications  root  causes 
are  never  quite  the  same  from  organization  to  organization. 
Analyzing  communications  patterns,  decision-making  pro¬ 
cesses,  and  detailed  planning  becomes  a  key  to  solutions  for 
the  organization. 

We  also  have  achieved  valuable  insights  from  APTW  struc¬ 
tured  Interviews  and  surveys  on  how  to  achieve  solid  success 
in  program  execution. 

Lesson  Learned  No.  7:  PMs'  goals  need  to  be  viewed  from  the 
implementer's  perspective  ("a  view  from  the  deck  plates")  to 
thoughtfully  build  a  practical,  executable  plan. 

Lesson  Learned  No.  8:  Managing  the  internal  and  external 
program  success  expectations  must  be  an  integral  part  of  any 
successful  acquisition  strategy. 

The  DAU  APTW  team  often  talks  with  and  surveys  the  con¬ 
tractor,  stakeholders,  and  others  in  the  decision  chain.  It's 
very  common  for  the  program  team  to  have  pockets  of  mis¬ 
understanding  or  lack  of  trust  that  need  fixing.  Quality  of  data 
sharing  can  be  "all  over  the  map"  within  the  government,  with 
disconnects  between  the  government  team  and  the  prime 
contractor  and  between  the  primes  and  their  subcontractors. 

Lessons  Learned;  Encourage  transparency  throughout  the 
extended  program. 


Early  program  data  transparency  from  the  start  improves 
the  quality  of  day-to-day  management  in  areas  as  various 
as  processing  CDRLs  and  drawings,  system  engineering 
reviews,  program  progress  assessments,  responding  con¬ 
sistently  to  requests  from  external  stakeholders,  and  many, 
many  more. 

The  APTW,  and  organizational  "deep  dives,"  are  among  DAU's 
most  complex  workshops.  Most  of  their  supporting  tools  can 
be  adapted  for  short  assist  visits— e.g.,  strategic  workshops  for 
PMs.  In  those  short  assists,  we  seek  to  understand  the  PMs' 
and  leaders'  goals,  interview  the  teams,  and  build  quick  reaction 
workshops.  Surveys  (we  have  a  large  database  of  survey  ques¬ 
tions  developed  from  looking  at  many  programs  in  different  life- 
cycle  phases)  can  help  a  program  office  analyze  organizational 
issues  or  internal  issues.  The  surveys  also  may  be  tailored  to 
analyze  specific  program  activities.  Program  office  interviews 
and  short  workshops  can  help  identify  the  need  for  program 
office  streamlining  or  issues  in  preparing  for  a  milestone.  In  sup¬ 
port  of  Better  Buying  Power,  we  also  are  aiding  programs  in  the 
"how  to"  for  implementing  BBP  initiatives. 

This  DAU  mission  assistance  toolkit  is  focused  on  helping  ac¬ 
quirers  and  their  organizations  adapt  to  program  changes  in 
our  dynamic  acquisition  world.  If  we  can  be  of  help,  please  give 
us  a  call! 


The  authors  can  be  contacted  at  John. Higbee(§)dau. mil  and  Jesse. 
Stewart@dau.mil. 


Defense  Acquisition 
Portal 

Online  Performance  Support  for  the 
Acquisition  Professional 

It's  a  single  point  of  entry  for  applications  in  the  Acquisition 
Knowledge  Management  System,  expanding  upon  and  replacing  the 
Acquisition  Knowledge  Sharing  System. 

You  can  use  the  Defense  Acquisition  Portal  to: 

■  Meet  your  training,  education,  and  on-the-job  support  needs 

■  Address  the  elements  and  forces  of  the  Big  Acquisition  process 

•  Gain  information  relevant  to  the  DoD  workforce  and  Industry 
partners,  through  execution  of  the  Big  Acquisition  process 
acquisition  policy 

•  Receive  support  throughout  the  milieu  of  the  acquisition  process 

•  Search  and  find  information  with  the  ACQuire  Search  format! 


^  Defense  Acquisition  Portal 

\  //  Acquisition  •  Requirements  •  PPBE 


Start  using  the  Defense  Acquisition  Portal  today! 

https://dap.dau.mil 


9 


Defense  AT&L:  May-June  2013 


Bridging  the  Gap 

Dedicated  Technology  Transition  Programs 
Accelerate  Technology  Adoption 


Brad  Pantuck 

ied  technology  transition  pro- 
can  be  highly  effective  and  ef- 
at  moving  technologies  across 
alley  of  death"  from  technology 
ers  to  acquisition.  The  programs 
'ork  best  do  this  by  facilitating 
alignment  among  the  key  stakehold¬ 
ers  (developers,  acquisition  officials, 
resource  sponsors,  and  users)  and 
requiring  a  short  timeline  for  comple¬ 
tion,  typically  2  or  3  years.  By  imple¬ 
menting  these  and  a  few  other  best 


Pantuck  is  a  technology  transition  manager  for  RAE,  LLC.  He  focuses  on  building  the  partnerships  and  setting 
up  the  processes  necessary  to  accelerate  technology  adoption. 
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practices,  dedicated  transition  programs  can  produce  high 
success  rates  that  are  essential  for  our  nation  to  keep  its 
technical  edge  and  save  operational  costs  during  a  period  of 
constricting  budgets. 

Why  Dedicated  Transition  Programs  Are 
Needed 

Three  primary  factors  reinforce  the  need  for  dedicated  tran¬ 
sition  programs.  First,  the  multiyear  acquisition  planning 
process  Is  inadequate  for  keeping  our  forces  a  step  ahead 
of  our  adversaries;  technology  changes  too  quickly,  and  new 
threats  emerge  every  day.  While  acquisition  programs  can 
and  do  integrate  new  technology,  all  too  frequently  the 
timelines  and  established  processes  of  the  acquisition 
system  prevent  new  ideas  that  can  improve  capability  or 
reduce  operational  costs  from  getting  into  the  hands  of  the 
warfighters  in  a  timely  fashion. 

Another  contributing  factor  addresses  the  "last  hurdle"  to 
adopting  technology.  Whether  one  is  refining  technology  from 
military  science  and  technology  investments  or  adapting  com¬ 
mercially  available  technology,  some  level  of  maturation,  test¬ 
ing,  certification  and/or  integration  often  is  needed  to  trans¬ 
form  technologies  Into  useful  military  products  and  to  ensure 
that  the  products  successfully  make  it  to  operational  users. 

In  addition  to  fielding  technologies  sooner,  focused  technology 
transition  programs  can  be  very  cost  efficient.  First,  a  short 
time  horizon  (3  years  or  less)  reduces  the  risks  of  require¬ 
ment  changes  and  technology  obsolescence— increasing  the 
likelihood  that  the  technology  will  be  fielded.  Expeditious  in¬ 
sertion  also  allows  technologies  intended  to  save  money  to 


achieve  operational  cost  savings  sooner.  Lastly,  because  the 
funding  for  each  effort  in  such  programs  is  typically  less  than 
a  few  million  dollars,  the  cost  of  failure  is  cheap.  While  success 
rates  vary,  in  the  best-managed  dedicated  technology  transi¬ 
tion  programs  more  than  70  percent  of  the  prototypes  are  in 
acquisition  or  fielded  within  3  years  of  initial  funding. 

What  Transition  Programs  Accomplish 

Dedicated  transition  programs  often  are  focused  on  either 
individual  (stand-alone)  devices  or  improving/replacing  one 
piece  of  a  larger  platform  or  system.  The  output  therefore 
Is  not  a  tank  but  an  improved  turret  rotation  motor;  not  an 
aircraft  carrier  but  a  high  temperature-resistant  coating  for 
aircraft  carriers'  flight  decks;  not  an  F/A-18  but  an  onboard 
high-speed,  large  bandwidth  network  to  connect  an  F/A-18's 
computer  systems. 

To  help  new  technologies  cross  the  finish  line,  OSD  and  the 
various  Services  designate  funds  for  technology  transition. 
According  to  the  Small  Business  Technology  Council  of  the 
National  Small  Business  Association,  there  are  almost  50 
technology  transition  funding  programs  within  the  Defense 
Department,  with  20  of  those  programs  oriented  toward  ac¬ 
celerating  transition.  Some  of  these  programs  are  focused  on 
transitioning  technology  originating  in  military  S&J  programs; 
others  are  focused  on  adapting  commercial  technology;  a  few 
are  agnostic  regarding  the  technology  source.  Regardless  of 
the  technology's  origin,  each  program's  desired  outcome  Is 
that  better  and/or  cost-saving  technologies  are  quickly  in¬ 
tegrated  into  end  users'  operations— expeditious  fielding  of 
technologies  addresses  critical  capability  shortfalls  that  can 
result  in  loss  of  life  and/or  failed  missions. 


One  example  of  a  suc¬ 
cessful  Department  of 
the  Navy  short-term  tran¬ 
sition  effort  is  a  gearbox 
repair  technology  for 
AH-1  helicopters  (see 
photo).  It  was  funded  by 
the  Navy's  Technology 
Insertion  Program  for 
Savings  In  FY  2011  ($1.8 
million)  and  fielded  at  the 
beginning  of  FY  2013.  Be¬ 
fore  this  new  cold-spray 
technology  was  transi¬ 
tioned,  abraded  AH-1 
combining  gearbox  hous¬ 
ings  would  be  transferred 
to  the  depot  for  repairs 
and  at  least  50  percent 
would  end  up  scrapped. 
This  technology  now 
enables  maintenance 

An  operator  demonstrates  repair  of  an  AH-1  helicopter  combining  gearbox  housing  using  cold  spray  tech-  personnel  to  quickly  and 
nology,  which  enables  repairs  closer  to  the  field.  This  will  save  the  Navy  $39  million  over  the  next  7  years.  cheaply  repair  gearboxes 
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Figure  1.  Transition  Venues 
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Figure  1  highlights  some  of  the  programs  the  Department  of  the  Navy  (DoN)  uses  to  transition  technology  to  operational  users.  These 
programs  typically  attempt  to  advance  technology  from  a  Technology  Readiness  Level  (TRL)  of  5  or  6  to  a  TRL  of  at  least  7  or  8,  a  ma¬ 
turity  level  that,  in  general,  presents  acceptable  technical  risk  for  the  acquisition  community  and  is  often  achievable  within  2  to  3  years. 


closer  to  the  field,  decreasing  scrap  rates  and  increasing  op¬ 
erational  readiness.  Originally  anticipated  to  save  the  Navy  $18 
million.  It  Is  now  projected  to  save  $39  million  over  the  next  7 
years— a  significant  return  on  investment. 

Another  Navy  transition  project  that  targets  an  immediate 
problem  is  the  Composite  Patch  Technology  for  Aluminum 
Structure  Repair.  The  Navy  faces  substantial  maintenance 
costs  associated  with  stress-corrosion  cracking  In  aluminum 
ship  superstructures.  The  fiber-reinforced  bonded  patch  that 
will  be  transitioned  through  this  effort  will  seal  cracks  and 
provide  structural  support  to  resist  further  crack  growth. 
When  fully  implemented,  this  technology  is  projected  to 
reduce  maintenance  costs  by  $30  million  across  the  CG-47 
ship  class  within  5  years,  compared  to  the  crack  welding  ap¬ 
proach  currently  used.  The  cost  to  transition  this  technology 
is  $1.7  million. 

The  Future  of  Transition  Programs 

Recent  appropriations  decisions  Indicate  a  renewed  focus  on 
transition.  For  example.  Congress,  through  the  National  De¬ 
fense  Authorization  Act  for  FY  2011,  created  the  Rapid  Innova¬ 
tion  Program  (known  within  DoD  as  "Rapid  Innovation  Fund"), 
which  focuses  on  transitioning  technologies  from  industry  into 
military  systems  within  2  years.  The  National  Defense  Autho¬ 
rization  Act  of  2012  contains  provisions  intended  to  increase 
the  number  of  Small  Business  Innovation  Research  (SBIR) 
Phase  III  contracts  issued,  the  final  phase  of  the  SBIR  program 
which  leads  to  transition.  The  emphasis  on  transition  Is  not 
constrained  to  the  military;  the  civilian  sector  recently  adopted 
similar  initiatives  aimed  at  fielding  technology  sooner.  In  the 


past  few  years.  Department  of  Homeland  Security's  (DHS) 
Science  and  Technology  Directorate  (S&T)  has  emphasized 
Its  Apex  program,  which  aims  to  quickly  transition  high-impact 
technologies  to  DHS  Components. 

Given  current  environmental  factors,  there  likely  will  be 
some  changes  to  transition  programs.  With  the  wars  in  Iraq 
and  Afghanistan  drawing  down,  some  programs  may  take  a 
longer-term  view,  focusing  on  fewer  technologies  with  a  larger, 
longer-term  impact.  Given  the  need  to  cut  overall  costs,  some 
transition  programs  may  emphasize  cost-savings  technolo¬ 
gies  over  those  that  increase  capability.  Because  private  sec¬ 
tor  developments  outpace  those  of  the  government  in  areas 
such  as  consumer  electronics,  cyber-security  technologies, 
and  information  technology,  some  transition  programs  may 
focus  more  on  adapting  technology  originated  outside  of  the 
government.  Nevertheless,  dedicated  transition  programs  will 
continue  to  play  a  key  role  in  fielding  technologies. 

Elements  of  a  Successful  Transition  Program 

The  biggest  challenge  in  technology  transition  is  stakeholder 
alignment.  The  key  partners  in  any  technology  transition  ef¬ 
fort-developers,  acquisition  officials,  and  users— have  dif¬ 
ferent  cultures  and  incentives.  Developers  are  incentivized 
toward  optimism  and  risk  taking,  while  acquisition  officials  are 
less  risk  tolerant  and  are  driven  by  cost,  performance,  and 
schedule  objectives.  Developers  tend  to  think  In  long  time  ho¬ 
rizons,  while  acquirers  have  firm  deadlines.  Users  are  much 
more  interested  in  practical  utility  than  in  technical  sophistica¬ 
tion  and  are  concerned  with  having  sufficient  units  available  for 
deployment  in  the  near  term.  "Wonderful"  technology  in  some 


13 


Defense  AT&L:  May-June  2013 


Figure  2.  Stakeholder  Alignment 
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Transition  Coordinators  align  the  stakeholders  to  quickly  produce  a  fielded  product. 


stakeholders,  creating  cohe¬ 
sion  and  accountability  (see 
Figure  2). 

Another  best  practice  is  to 
spend  time  and  resources 
aligning  the  stakeholders  early 
in  the  process.  From  the  start, 
it  should  be  clear  that  warfight¬ 
ers  need  the  new  technology, 
the  acquisition  community 
wants  to  buy  it,  the  resource 
sponsor  has  the  funds  to  pay 
for  it,  and  the  engineers/ven¬ 
dors  can  build  it.  Proper  co¬ 
ordination  at  the  early  stages 
of  transition  helps  developers 
avoid  successfully  demonstrat¬ 
ing  a  technology  only  to  find 
that  neitherthe  acquisition  nor 
user  communities  are  prepared 
to  accept  it. 


distant  future  has  less  value  to  the  warfighter  than  "good" 
technology  today. 

Successful  technology  transition  organizations  address  this 
cultural  diversity  through  a  variety  of  "best  practices."  One 
of  the  most  useful  is  to  employ  an  independent  team  of 
coordinators  who  facilitate  communications  and  reconcile 
differences  among  the  disparate  stakeholder  communities. 
In  such  a  coordinator  or  "relationship  manager"  model, 
these  individuals  guide  the  movement  of  technology  from 
the  development  phase  into  the  acquisition  and  produc¬ 
tion  phase.  At  the  beginning  of  the  process,  they  work  with 
developers  to  articulate  a  technology's  business  case,  just 
as  venture  capitalists  do  with  entrepreneur  principals  in 
startups.  Then,  they  conduct  the  necessary  technical,  busi¬ 
ness,  and  programmatic  due  diligence  to  raise  acquisition 
customer  confidence  and  reduce  risks  to  transition.  Transi¬ 
tion  coordinators  also  establish  resource  sponsor  and  user 
buy-in,  and  facilitate  and  document  agreements  among  the 


Stakeholder  engagement  should  culminate  in  a  technology 
transition  agreement  (TTA),  signed  prior  to  the  project's  fund¬ 
ing.  The  TTA  describes  the  transition  path  and  codifies  the 
partners'  agreements,  binding  them  together  for  a  common 
purpose.  It  typically  includes  the  following  components: 

Technology  Opportunity  and  Business  Case;  A  description 
of  the  technology  to  be  transitioned,  including  the  scientific 
basis,  the  maturity  of  the  technology,  and  how  the  technology 
will  fit  into  any  larger  system.  The  business  case  presents  the 
reasons  for  the  acquisition,  resource,  and  user  communities' 
compelling  interest  in  obtaining  the  technology,  often  by  de¬ 
scribing  the  comparative  benefits  of  the  technology  in  refer¬ 
ence  to  alternate  or  emerging  technologies  in  the  same  area. 
Focusing  on  one  technical  goal  per  agreement  is  an  important 
way  to  minimize  technical  risk. 

Scope  of  Work  and  Risks;  A  detailed  list  of  the  tasks  to  be 
performed,  along  with  the  attendant  roles  and  responsibilities. 


With  the  wars'in  Iraq  and  Afghanistan  drawing  down,  some 


programs  may  take  a  longer-term  view,  focusing  on  fewer 
technologies  with  a  largerTfon^r-term  impact. 
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A  composite  patch  repairs  a  CG-47  class  ship's  cracked  aluminum  superstructure.  If  successful,  it  will  save  $30  million  over  5  years. 


This  ensures  that  a  complete  solution  is  achieved  and  supports 
resource  project  planning  and  management.  Identification  of 
the  risks  (technical,  business,  and  programmatic)  educates 
the  acquisition  and  resource  decision  makers  and  provides  a 
basis  for  risk  mitigation  plan  development. 

Recipient  and  Acquisition  Cost:  The  organization  and  indi¬ 
viduals  that  will  receive  the  technology  and  their  out-year  in¬ 
tegration  and  sustainment  funding  costs  once  the  technology 
is  transitioned.  This  allows  the  customer  to  plan  ahead  and 
budget  for  receiving  the  technology. 

Milestones;  Key  events  and  dates  that  are  identified  to  align 
the  stakeholders  and  to  provide  for  accountability  and  "off 
ramps"  during  the  course  of  the  project. 

Seminal  Transition  Event  and  Metrics;  A  clear  end  point  for 
the  engineers  who  develop,  integrate,  and  test  the  technol¬ 
ogy.  Making  the  acceptance  criteria  transparent  from  the  start 
reduces  the  risk  that  the  approval  authority  will  change  its 
mind  midstream. 

Signatures  from  the  Partner  Organizations;  A  TTA 

signed  by  senior  decision  makers  who  are  able  to  make 
commitments  on  behalf  of  their  organizations.  The  TTA's 
goal  is  not  to  hold  the  organization  legally  accountable. 


but  to  drive  awareness  and  commitment.  If  any  one  of 
the  partners  (developers,  users,  acquisition  official,  and 
resource  sponsors)  waivers  in  commitment,  the  agree¬ 
ment  provides  a  basis  for  reengagement. 

Once  the  TTA  is  signed,  successful  transition  programs  apply 
resources  to  monitoring.  Transition  coordinators  identify  and 
mitigate  risks  and  obstacles  (before  they  become  roadblocks) 
on  the  path  toward  acceptance  by  the  acquisition  community 
and  adoption  by  the  user  community.  If  milestones  are  missed 
or  the  receiving  program's  plans  change  such  that  the  transi¬ 
tion  cannot  be  completed  on  time,  the  transition  program  can 
pull  back  remaining  funds  and  reassign  them  to  a  project  that 
will  transition. 

Conclusion 

Successful  transition  programs  align  the  key  stakeholders  to 
accelerate  the  adoption  of  new  or  cost-savings  technologies. 
By  increasing  the  speed  and  efficiency  with  which  science  and 
technology  investments  are  exploited,  they  make  maximum 
use  of  limited  funding,  a  quality  all  the  more  important  to  our 
warfighters  and  nation  at  a  time  when  resources  are  more 
constrained  and  every  dollar  must  count. 


The  author  can  be  contacted  at  bpantuck@rae-llc.com. 
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Contract  Oversight 
in  a  Contingency  Environment 

We  Bought  It,  You  Own  It 


Maj.  James  E.  Thomas,  USAF 


uring  my  latest  deployment  in  Afghanistan,  I  led  a  Joint  office  consisting  of  Air  Force, 
Army  and  Navy  personnel  (active  duty,  reservists.  Defense  Department  civilians, 
and  contractors)  as  NATO  Training  Mission-Afghanistan/Combined  Security  Transi¬ 
tion  Command-Afghanistan's  (NTM-A/CSTC-A))  Contract  Management  Oversight 
(CMO)  Office. 

The  office  was  stood  up  in  April  2010  to  rectify  multiple  Inspector  General  (IG)  and  Government  Accountability 
Office  (GAO)  reports  indicating  a  lack  of  "hands  on  oversight"  for  contracts  throughout  the  Afghanistan  The¬ 
ater.  We  were  charged  by  senior  leadership  to  ensure  that  "contract  owners"  provide  effective  management 


Thomas  is  a  major  in  the  US.  Air  Force  and  a  professor  of  acquisition  management  at  Defense  Acquisition  University's  Midwest  Region  in 
Kettering,  Ohio. 
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and  oversight  of  more  than  340  service  contracts  and  1,000 
construction  contracts  with  total  value  in  excess  of  $5  billion 
(FY  2011  data).  Basically,  our  main  focus  was  to  ensure  others 
were  doing  their  job  in  evaluating  contractor  performance  and 
to  provide  assistance  and  guidance  when  and  where  neces¬ 
sary.  Pretty  clear,  executable  guidance— right? 


NTM-A/CSTC-A  contracts  can  be  commonly  referred  to  as 
"local"  or  "CONUS."  The  term  "local"  applies  to  contracts 
awarded  by  CENTCOM  Contracting  Command's  RCCs  op¬ 
erating  in  Afghanistan.  The  term  CONUS  applies  to  all  ASFF 
contracts  awarded  in  the  United  States,  usually  via  a  Pseudo- 
Foreign  Military  Sales  case  with  execution  in  Afghanistan. 


The  Afghan  National  Security  Forces  (Afghan  National  Army, 
Afghan  National  Police)  rely  heavily  on  contractor  support 
for  many  equipping/training/sustaining  functions  such  as 
facility  maintenance,  construction,  combat  skills  training, 
life  support,  and  vehicle/weapon  procurement  and  main¬ 
tenance.  While  Headquarters  NTM-A/CSTC-A  provides 
funding  for  these  efforts  through  the  Afghan  Security  Forces 
Fund  (ASFF),  Regional  Command  (RCs),  Regional  Support 
Commands  (RSCs),  Headquarters  Directorates,  and  other 
organizations  generate  requirements,  "own  the  contract," 
and  are  responsible  for  providing  stewardship  and  oversight 
of  contracts  funded  with  ASFF. 

RCs,  RSCs,  and  Headquarter  Directorates  and  any  other  orga¬ 
nizations  using  ASFF  are  required  to  ensure  effective  contract 
execution  and  surveillance  by  assigning  an  adequate  number 
of  trained  contracting  officer  representatives  (CORs)  to  con¬ 
duct  hands-on  audits  measuring  contractor  performance. 
Again,  sounds  easy ...  but  in  a  contingency  environment,  with 
personnel  rotating  in  and  out  on  a  daily  basis,  with  restricted 


Once  a  contract  was  identified,  CMO  personnel  would  obtain 
a  copy  (harder  to  do  than  it  sounds),  read  the  contract  to  find 
clues  as  to  who  was  the  initiating  organization  and  then  asso¬ 
ciate  the  contract  to  a  Regional  Command,  Regional  Support 
Command  or  Directorate.  This  led  to  the  development  of  a 
database ,  which  became  the  authoritative  source  for  tracking 
status  and  reporting  to  the  three-star  NTM-A/CSTC-A  com¬ 
mander,  and  also  provided  Information  to  numerous  watch¬ 
dog  agencies  (IG,  GAO,  Commission  for  Wartime  Contracting, 
etc.),  that  are  involved  in  reporting  status  to  Congress. 

The  database  included  contract  number,  dollar  amount  as¬ 
sociated  with  execution  year  and  options,  points  of  execu¬ 
tion,  contracting  officer  and  surveillance  personnel,  and 
audit  dates  along  with  many  other  data  points.  This  tool 
and  the  person  who  created  it  were  amazing  as  it  would 
create  stoplight  charts  reflecting  number  of  contracts,  audit 
complete  percentages,  status  of  surveillance  personnel  (i.e., 
present/departed/departing)  and  contract  status  (active/ 
expired/expiring). 


lovement 


In  a  contingency  environment,  with  personnel  ro! 
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communication,  where  douars  flow  almost  unfetter^ 
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guidance"  becomes  increasingly  difficuit? 
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movement  and  communication,  where  dollars  flow  almost 
unfettered,  and  with  an  enemy  that  is  not  concerned  about 
whether  monthly  audits  are  completed,  execution  of  what  Is 
"pretty  clear  guidance"  becomes  Increasingly  difficult. 

The  Contract  Management  Office  (CMO)  attacked  the  dif¬ 
ficult  problem  of  tracking  new  and  existing  contracts,  requir¬ 
ing  using  organizations  to  take  ownership  and  stewardship  of 
those  contracts,  and  reporting  progress  to  senior  leadership 
using  a  three-pronged  approach.  First,  a  group  of  hard-work¬ 
ing,  smart,  and  dedicated  professionals  who  preceded  me  in 
theater  undertook  the  Herculean  task  of  identifying  existing 
local  and  Continental  U.S.  (CONUS)  contracts  by  working 
with  Central  Command  (CENTCOM)  Regional  Contracting 
Centers  (RCCs),  using  organizations  throughout  Afghani¬ 
stan  and  CONUS  Contracting  Centers  stateside.  In  general. 


The  database  and  associated  stoplight  charts  became  a  tool 
for  holding  commanders  and  directors  (contract  owners)  ac¬ 
countable  for  contract  execution  and  surveillance.  Stoplight 
charts  were  displayed  at  the  three-star's  staff  meetings  and 
commanders/directors  were  afforded  the  opportunity  to 
explain  status  (green=good,  red=bad).  Lastly,  educating  an 
ever-changing  cast  of  leaders  and  surveillance  personnel  at  the 
point  of  execution  became  our  biggest  challenge.  We  spent 
many  hours  on  the  phone  and  traveling  throughout  the  theater 
to  help  commanders  and  CORs  understand  the  multiple  levels 
of  contracting  activity  within  their  "battle  space." 

One  of  the  many  challenges  we  faced  was  a  lack  of  situ¬ 
ational  awareness  on  the  part  of  regional  commanders  and 
staff  directors.  In  some  cases,  these  leaders  simply  didn't 
know  that  in  taking  the  lead  of  an  organization,  they  might 
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in  fact  be  taking  responsibility  for  cost,  schedule,  and  perfor¬ 
mance  of  dozens  of  multimillion-dollar  contracts.  This  can  be 
frustrating  for  the  leader  as  this:  (a)  may  come  as  a  surprise; 
(b)  taxes  finite  manpower  resources;  and  (c)  drives  reporting 
requirements  that  may  seem  to  fall  outside  the  normal  chain 
of  command.  A  case  in  point:  A  new  Army  general  officer 
takes  over  a  Regional  Command.  He  or  she  is  overseeing 
a  Relief  in  Place/Transfer  of  Authority  (RIP/TOA).  His  or 
her  priorities  are  surely  focused  on  managing  hundreds  of 
troops  by  providing  transport,  shelter,  security  and  basic  life 


Another  aspect  of  training  included  educating  COR  person¬ 
nel.  While  there  are  mandatory  courses  required  prior  to  be¬ 
coming  a  COR,  this  training  in  and  of  itself  does  not  prepare 
someone  to  function  efficiently  as  a  COR.  Contracting  offi¬ 
cers  provide  contract-specific  training,  to  include  how  to  fill 
out  an  audit  form  and  explaining  contract  requirements,  but 
in  reality  a  "qualified"  COR  has  to  have  a  deep  understanding 
of  why  a  contract  is  in  place,  the  technical  issues  associated 
with  its  execution,  as  well  as  a  clear  picture  of  the  end  state. 
All  this  is  required  while  the  COR  keeps  the  contractor  at 
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understanding  of  why  a  contract  is  in  place, 
technical  issues  associated  with  its  executSyas*' 
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support,  executing  the  mission,  etc.  One  of  our  jobs  was  to 
ensure  these  leaders  understood  for  which  contracts  they 
were  responsible  and  to  help  them  develop  an  effective  over¬ 
sight  and  reporting  system. 

As  part  of  the  reporting  process,  commanders  and  direc¬ 
tors  were  required  to  brief  the  NTM-A/CSTC-A  deputy 
commander  for  programs  (the  position  that  manages  all 
ASFF  for  the  command)  on  the  execution  of  each  of  their 
contracts.  They  would  have  to  answer  tough  questions 
regarding  contract  performance,  contract  effectiveness, 
and  cost  effectiveness.  Examples  of  such  questions:  Is  this 
contract  relevant  to  the  mission  as  it  exists  today?  Is  the 
contractor  doing  what  we've  paid  him  to  do?  Are  we  paying 
the  contractor  to  do  the  right  thing?  To  what  extent  is  the 
customer  satisfied?  Are  we  getting  expected  value  from  the 
contract  relative  to  cost?  Is  the  contract  worth  the  invest¬ 
ment  relative  to  cost?  How  effective  is  the  contractor  at 
fulfilling  his  requirements?  Do  our  requirements  still  exist? 
How  do  you  measure  success? 

This  in  and  of  itself  created  a  threefold  problem  for  command¬ 
ers  and  directors: 

■  They  had  to  take  a  hard  look  and  dive  deep  into  contract 
requirements  and  contractor  performance. 

■  In  most  cases,  the  Deputy  Command  for  Programs  does  not 
fall  within  the  operational  control  or  administrative  control 
of  the  regional  commanders/directors. 

■  In  many  cases,  the  commander/director  outranked  the 
deputy  commander  for  programs  (then  a  colonel  filling  a 
brigadier  general  position).  In  all  cases,  CMO  personnel 
engaged  to  work  through  issues,  soothe  egos,  and  educate 
personnel. 


arm's  length  so  personal  bias  does  not  interfere  with  effective 
performance  evaluation. 

CMO  personnel  wrote  a  Standard  Operating  Procedure  imple¬ 
mented  throughout  the  theater  outlining  roles  and  responsi¬ 
bilities  for  both  pre-award  and  post-award  contracting  phases 
to  help  educate  senior  leadership  and  surveillance  personnel. 
We  also  spent  a  lot  of  "one-on-one  time"  talking  about  how  to 
form  a  multifunctional  requirements  development  team,  how 
to  develop  an  executable  Performance  Work  Statement  and 
a  Quality  Assurance  Surveillance  Plan,  and  how  to  effectively 
organize  surveillance  personnel  to  measure  performance  and 
effectiveness.  This  was  an  ongoing  process  because,  as  stated 
before,  personnel  were  rotating  in  and  out  constantly. 

Another  issue  was  a  lack  of  experienced  personnel  to  con¬ 
duct  surveillance  at  the  point  of  contract  execution.  In  many 
cases,  a  contractor  may  be  working  at  hundreds  of  different 
locations  to  fulfill  contract  requirements.  For  instance,  our 
language  training  contract  had  in  excess  of  100  points  of 
execution  throughout  theater.  We  hired  a  group  of  Red  River 
Army  Depot  personnel,  trained  them,  and  assigned  them  to 
RCs  and  RSCs  to  become  full-time  CORs.  This  fact,  coupled 
with  bringing  on  experienced  former  government  contract¬ 
ing  officers  to  help  requesting  activities  generate  solid  re¬ 
quirements  documents,  aided  both  contract  execution  and 
performance  measurement. 

At  the  end  of  my  tour,  I  was  proud  of  the  hard  work  we  had 
done  and  confident  that  those  who  followed  would  continue 
our  work  of  holding  requiring  units  accountable  for  effectively 
managing  contractor  performance. 


The  author  can  be  contacted  at  JamesThomas^dau.mil. 
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Determining  the  Probable  Cost 


our  organization  has  just  issued  a  Request  for  Proposal  (RFP),  and,  in  response,  you 
have  received  several  proposals.  In  your  RFP,  you  stated  that  the  government  was  con¬ 
templating  the  award  of  a  cost-reimbursement  contract. 


Y 

I  You  are  preparing  to  perform  your  analysis.  Before  starting,  you  go  to  the  Federal  Acquisition  Regulation 
(FAR),  specifically  FAR  Part  15.404-l(d),  and  realize  that  the  FAR  requires  you  to  perform  cost  realism 
analysis  to  determine  the  probable  cost  of  performance  for  each  offeror.  You  start  asking  yourself  a  series  of  ques¬ 
tions  such  as:  What  is  cost  realism  analysis?  When  does  cost  realism  need  to  be  done?  How  do  I  determine  the 
probable  cost?  What  resources  are  available  to  assist  me  in  developing  a  probable  cost?  Does  the  government  get 
many  protests  regarding  cost  realism  analysis?  It  is  hoped  that  this  article  will  help  answer  these  questions  and  more. 


Nkolella  is  a  professor  of  contract  management  at  the  Defense  Acquisition  University's  South  Region  in  Huntsville,  Ala. 
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Per  FAR  2.101  and  15.404-l(d)  and  Contract  Pricing  Ref¬ 
erence  Guides  (CPRG),  Volume  4,  Chapter  8,  Paragraph 
8.1,  cost  realism  analysis  is  "the  process  of  independently 
reviewing  and  evaluating  specific  elements  of  each  of¬ 
feror's  proposed  cost  estimate  to  determine  whether  the 
estimated  proposed  cost  elements  are  realistic  for  the  work 
to  be  performed;  reflect  a  clear  understanding  of  contract 
requirements;  and  are  consistent  with  the  unique  methods 
of  performances  and  materials  described  in  the  offeror's 
technical  proposal." 

Let's  dissect  the  above  definition  a  little  bit  more  by  focusing 
on  several  key  terms.  First,  it  is  an  "independent  process," 
which  means  that  as  a  contracting  professional  you  have  to 
do  the  reviewing  and  evaluating.  This  does  not  mean  you 
cannot  solicit  input  or  help  from  other  government  person¬ 
nel  (contracting  officer  representatives— CORs;  Technical 


Now  that  we  know  what  cost  realism  analysis  is,  we  need  to 
answer  the  next  question:  When  does  it  need  to  be  done? 
FAR  Part  15.404-l(d)(2)  states  that  cost  realism  analysis  shall 
be  performed  on  cost-reimbursement  contracts  to  determine 
the  probable  cost.  All  contracting  professionals  should  know 
that  the  word  "shall"  means  "must."  So  as  a  contracting  pro¬ 
fessional,  you  must  perform  cost  realism  analysis  on  all  cost- 
reimbursement  contracts.  It  does  not  get  any  clearer  than  that. 

The  next  two  questions— "How  do  I  determine  the  probable 
cost,  and  what  resources  are  available  to  assist  me  in  develop¬ 
ing  a  probable  cost?"— kind  of  go  hand  and  hand  because  you 
cannot  do  one  without  doing  the  other.  Knowing  what  sources 
of  information  are  available  to  you  when  trying  to  determine 
the  probable  cost  will  make  your  job  much  easier.  As  a  govern¬ 
ment  contracting  professional,  there  are  numerous  sources  of 
information  you  can  use  to  help  you  determine  the  probable 


Knowing  what  sources  of  information  are  available 
to  you  when  trying  to  determine  the  probable  cost 
will  make  your  job  much  easier. 


Points  of  Contact— TPOCs;  engineers,  etc.),  or  agencies  like 
the  Defense  Contract  Audit  Agency  (DCAA).  However,  it 
does  mean  that  you,  the  contracting  officer,  will  make  the 
judgment  independent  of  any  of  the  before  mentioned  per¬ 
sonnel  or  agencies.  Second,  cost  realism  analysis  includes  the 
"reviewing  and  evaluating  process  of  specific  elements."  The 
elements  to  which  this  definition  refers  are  cost  elements 
as  defined  in  FAR  Part  15.408,  Table  15-2,  Roman  Numeral 
II  (Cost  Elements),  such  as  direct  labor,  indirect  costs,  other 
costs,  etc.  Does  that  mean  you  have  to  look  at  each  cost 
element  when  performing  cost  realism  analysis?  Not  neces¬ 
sarily.  If  a  cost  element  appears  reasonable  based  on  your 
preliminary  review  and  analysis,  you  may  not  have  to  ana¬ 
lyze  it  any  further.  Also,  reviewing  and  evaluating  specific 
cost  elements  can  be  limited  to  substantial  costs  (Controller 
General  Case:  B-271302.2). 

Finally,  the  cost  realism  definition  further  states  that  in  doing 
your  analysis  you  must  look  at  each  "offeror's  proposed  cost 
estimate."  This  Is  important.  You  must  analyze  and  develop 
a  probable  cost  for  each  offeror.  You  cannot  simply  do  one 
analysis  and  one  probable  cost  and  apply  It  to  all  offerors' 
proposals.  If  you  did  this,  you  would  not  be  in  compliance 
with  FAR  Part  15.404-l(d)(2)  and  CPRG.  Also,  realize  that 
each  offeror  will  have  a  different  technical  approach  and  ac¬ 
counting  system,  so  using  a  single  probable  cost  and  applying 
it  across  the  board  to  all  proposals  would  be  impractical.  By 
defining  cost  realism  analysis  and  then  breaking  down  its  key 
terms,  we  were  able  to  answer  the  question  "What  is  cost 
realism  analysis?" 


cost.  Including  an  Independent  Government  Estimate  (IGE), 
cost  estimating  relationships,  wage  determinations,  technical 
evaluations,  audit  reports,  forward  pricing  rate  agreements 
(FPRA),  and  results  from  cost  estimating  system  reviews,  just 
to  name  a  few.  In  addition,  you  can  obtain  assistance  from 
other  members  of  the  government  acquisition  team  like  your 
technical  specialists  (CORs/TPOCs)  and  personnel  from 
both  DCAA  and  the  Defense  Contract  Management  Agency 
(DCMA).  Each  of  these  members  is  uniquely  qualified  to  assist 
you  in  evaluating  technical  and  pricing  proposals.  For  example, 
an  in-house  technical  expert,  COR,  can  provide  you  with  valu¬ 
able  input  regarding  how  realistic  an  offeror's  proposed  cost 
estimate  Is  with  regard  to  material  costs,  labor  mix,  and  labor 
hours.  DCAA  is  familiar  with  offerors'  accounting  systems  and 
indirect  rates  and  can  help  you  determine  if  indirect  rates  are 
significantly  lower  than  projected  rates.  DCMA  can  provide 
you  with  an  array  of  experts  (Quality  Assurance  Specialists, 
Engineers,  Cost/Price  Analysts,  Industrial  Specialists,  etc.), 
that  can  help  answer  any  questions  that  your  In-house  tech¬ 
nical  personnel  may  have  about  a  proposal.  DCMA  also  can 
help  answer  any  questions  regarding  FPRAs  or  Forward  Pricing 
Rate  Recommendations  (FPRRs). 

Table  1  shows  sources  and  resources  that  may  help  illustrate 
how  one  can  determine  the  probable  cost. 

FAR  Part  15.404-l(d)(2)(i)  states  that  the  probable  cost  may 
differ  from  proposed  cost  and  should  reflect  the  government's 
best  estimate.  Section  (ii)  of  the  same  reference  further 
states  that  the  probable  cost  Is  determined  by  adjusting  each 
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Table  1.  Match-up  of  Source  and  Resource  with  Cost  Elements 


Cost  Elements 

Proposed 

Probable  Cost 

Resources  and  Sources  Available 

1.  Material 

$13,000 

$10,000 

COR— Technical  Evaluation 

2.  Eng.  Direct  Labor 

$1,000,000 

$1,250,000 

Contract  Specialist— Wage  Determination 

3.  Eng.  OH 

$1,250,000 

$1,250,000 

DCMA— Forward  Pricing  Rate  Agreement  (FPRA) 

4.  ODC  (Travel) 

$2,000 

$2,000 

Contract  Specialist— Joint  Travel  Reg.  (JTR) 

5.  Subtotal  Production 

$2,265,000 

$2,512,000 

DCAA— Total  Cost  Input  or  Value  Added 

6.G&A 

$226,500 

$251,200 

Cost  Price  Analyst— Regression  Analysis 

Note:  Some  elements  may  have  more  than  one  source  or  resource. 


offeror's  proposed  cost  to  reflect  any  additions  or  reductions 
in  cost  elements  to  realistic  levels  based  on  the  cost  realism 
results.  As  you  can  see  from  the  above  diagram,  a  reduction 
to  the  cost  element  of  material  was  made  and  then  additions 
to  Engineering  Direct  Labor  and  general  and  administrative 
expenses  were  made.  The  former  reduction  was  made  based 
on  feedback  from  the  COR,  and  the  latter  additions  were  made 
based  on  Contract  Specialist  and  Cost  Price  Analysts  input. 

Other  sources  and  resources  were  used  to  evaluate  the  best 
value  of  the  remaining  cost  elements.  But,  since  these  cost 
elements  (Engineering  Overhead  [OH]  and  Other  Direct  Costs 
[ODC])  appeared  realistic,  the  contract  specialist  determined 
that  no  adjustments  were  necessary. 

The  process  outlined  above  would  need  to  be  repeated  in 
order  to  determine  the  government's  probable  cost  of  each 
offeror's  proposal.  This  simple  but  fairly  accurate  illustration 
demonstrates  the  process  a  contracting  professional  should 
go  through  when  trying  to  determine  the  probable  cost  of  per¬ 
formance.  In  our  illustration,  we  had  sources  and  resources 
identified  to  assist  us  in  determining  the  probable  cost,  but 
this  may  not  always  be  the  case.  FPRAs,  historical  data  for 
regression  analysis,  and  wage  determinations  may  not  always 
be  available  or  in  existence.  In  these  circumstances,  it  makes 
determining  the  probable  cost  more  difficult  but  not  impos¬ 
sible.  You  will  need  to  improvise  (use  other  methods)  to  de¬ 
termine  the  government's  probable  cost. 

So  at  this  point  some  of  you  are  undoubtedly  thinking  cost 
realism  analysis  and  especially  determining  the  probable  cost 
sounds  like  a  judgmental  process  and  must  lead  to  numer¬ 
ous  protests  filed  against  the  government.  This  takes  us  to 
our  last  question,  "Does  the  government  get  a  lot  of  protests 
regarding  cost  realism  analysis?"  The  cost  realism  analysis  bid 
protest  results  of  the  last  3  years,  listed  in  Table  2,  may  provide 
a  pleasant  surprise. 

According  to  the  Government  Accountability  Office  (GAO) 
official  website,  www.gao.gov/legal/bids/bidprotest.html, 
from  Jan.  1,  2010,  to  Dec.  31,  2012,  DoD  received  501  bid 
protests.  Of  the  501  protests,  only  42,  or  8.4  percent,  were 
cost  realism  related.  For  2012  and  the  17  protests  received 


(not  including  the  three  that  are  still  open),  the  sustained 
vs.  denied  ratio  was  one-fourteenth,  or  7  percent.  The  one 
protest  that  was  sustained  was  due  to  the  government  not 
following  one  of  the  cardinal  rules  of  FAR  15.404-1(d)  and 
Volume  4,  Chapter  8  of  the  CPRG.  Instead  of  developing  a 
probable  cost  for  each  offeror's  proposal,  the  agency  com¬ 
pared  one  offeror's  proposal  to  the  median  price  proposed 
by  other  offerors,  some  of  which  already  were  deemed  unac¬ 
ceptable  due  to  unreasonably  high  prices. 

This  rationale  was  unsound,  and  taking  a  one-size-fits-all  ap¬ 
proach  is  not  in  accordance  with  the  FAR  or  CPRG  and  can 
lead  to  a  protest  and  a  subsequent  victory  for  the  protester. 
However,  the  number  of  bid  protests  Is  remarkably  low  and 
indicates  that  the  majority  of  the  government  agencies  are 
performing  cost  realism  analysis  and  determining  the  probable 
cost  in  accordance  with  FAR  and  CPRG  guidance  and  solicita¬ 
tion  criteria. 

With  cost  realism  analysis  now  being  taught  in  the  contract¬ 
ing  curriculum  in  such  Defense  Acquisition  University  (DAU) 
courses  as  CON  170  (Fundamentals  of  Cost  and  Price  Analy¬ 
sis),  CON  270  (Intermediate  Cost  and  Price  Analysis),  and 
CON  280  (Source  Selection  and  Administration  of  Service 
Contracts)  and  with  more  government  contracting  profes¬ 
sionals  receiving  such  training  earlier  in  their  careers,  it  would 
be  reasonable  to  expect  the  number  of  cost  realism  analysis 
protests  to  steadily  decline  in  the  future. 

The  author  can  be  contacted  at  anthony.nicolella@dau.mn. 


Table  2.  GAO  Protests  Relating  to  Cost 
Realism,  DoD 


CY2010 

CY  2011 

CY  2012 

Total 

Breakdown  by 
Service 

8 

17 

17 

42 

Air  Force  - 13 
Army  - 10 

Navy  -  9 
Other  -  8 
Marines  -  2 

Current  as  of  11/08/2012 
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Leading  Complex  Projects 
in  the  DoD 


Steven  R.  Meier,  Ph.D. 


jajy's  projects  require  move 
to  handle  increased  com- 
irge  uncertainty.  Complex 
(oth  difficult  and  challeng- 
jhe  most  seasoned  project 
magers.  Leading  these  types  of  proj¬ 
ects  requires  a  versatile  skill  set,  the 
ability  to  manage  the  unforeseen,  and 
a  strategic  vision.  Complex  projects  re¬ 
quire  more  than  just  management;  they 
require  leadership. 


Meier,  Ph.D.,  Program  Management  Professional  (PMP),  has  more  than  20  years  of  federal  and  private  industry 
experience  focused  on  the  defense,  intelligence,  and  civil  aerospace  communities.  Meier  is  the  founder  ofSRM 
Consulting,  LLC,  a  consulting  firm  that  specializes  in  linking  strategy  and  execution  to  business  results.  He  was 
a  vice  president  at  the  Lockheed  Martin  Corp.,  and  a  member  of  the  federal  Senior  Executive  Service  at  NASA. 
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Leadership  is  important  in  every  project  but  can  be  even  more 
challenging  for  complex  projects  since  there  is  a  multitude  of 
variables  to  manage  all  at  once.  Complex  projects  lie  between 
traditional  project  management  and  extreme  project  manage¬ 
ment  and  they: 

■  Utilize  new  or  unproven  technology. 

■  Consist  of  independent,  interacting  elements  that  require 
integration. 

■  Involve  two  or  more  stakeholders. 

■  Entail  a  dynamic  human  resource  environment. 

These  traits  are  common  to  many  Department  of  Defense 
projects.  First,  most  DoD  projects  have  a  goal  of  demonstrat¬ 
ing  unproven  technology  to  meet  the  increasing  needs  of  the 
warfighter  or  to  address  a  new  threat.  Second,  in  most  cases, 
DoD  projects  Involve  the  designing,  building,  and  delivery  of 
a  system  or  subsystem  that  fits  into  a  larger  architecture  and 
requires  integration  at  multiple  levels.  Third,  in  these  times 
of  shrinking  budgets  and  affordability,  many  programs  have 
adopted  cost-sharing  partnerships  with  other  agencies  to 
ease  the  financial  burden.  And  fourth,  many  DoD  organiza¬ 
tions  involved  In  complex  project  developments  have  military 
and  civilian  personnel  who  rotate  every  2  to  3  years,  creating 
a  dynamic  human  resource  environment. 

Project  Leadership  Best  Practices 

The  purpose  of  this  article  Is  to:  (1)  add  to  the  existing  knowl¬ 
edge  base  of  best  project  management  leadership  practices, 
(2)  confirm  the  results  of  other  publications  and  studies  on 
complex  DoD  projects,  (3)  provide  seven  practices  for  leading 
complex  projects,  and  (4)  discuss  the  causes  of  unsuccessful 
complex  DoD  projects. 

Specifically,  this  article  identifies  seven  leadership  practices 
that  have  been  utilized  to  lead  complex  ground,  air,  and  space 
projects  to  successful  outcomes.  They  Include: 

■  Be  decisive. 

■  Battle  overzealous  advocates. 

■  Mature  new  technology  early  and  in  a  serial  process. 

■  Experiment  early  and  fail  early. 

■  Stop  requirements  creep. 

■  Take  great  care  in  managing  Interfaces. 

■  Create  a  software  integrated  product  team. 

In  the  remaining  sections  of  this  article,  I  will  discuss  each 
of  these  best  practices  in  detail  and  provide  data  to  support 
each  practice. 

Be  Decisive 

One  of  the  most  critical  roles  of  a  DoD  project  leader  is  to 
make  decisions.  To  a  large  degree,  the  success  of  any  project 
comes  down  to  project  personnel  making  good  decisions  on 
a  daily  basis.  Many  leaders  are  reticent  to  make  timely  deci¬ 
sions  for  fear  of  making  the  wrong  decision,  or  require  addi¬ 
tional  studies  to  provide  more  data  to  execute  a  decision.  The 


inability  to  make  timely  decisions  contrasts  to  past  and  current 
leaders  who  were  well  aware  of  the  critical  need  to  make  timely 
decisions.  To  quote  President  Theodore  Roosevelt,  "'In  any  mo¬ 
ment  of  decision,  the  best  thing  you  can  do  is  the  right  thing, 
the  next  best  thing  is  the  wrong  thing,  and  the  worst  thing  you 
can  do  is  nothing."  Moreover,  John  Chambers,  the  longstand¬ 
ing  and  admired  CEO  of  CISCO,  echoes  Roosevelt's  sentiments 
with,  "Without  exception,  all  of  my  biggest  mistakes  occurred 
because  I  moved  too  slowly." 

Decisions  need  to  be  made  in  a  timely  manner  to  keep  a  pro¬ 
gram  moving  forward  and  to  maintain  high  motivation  levels 
for  the  project  team.  Being  decisive  does  not  refer  to  making 
haphazard,  uninformed  decisions  but  making  decisions  that 
are  based  on  data,  facts,  and  experience.  Since  most  defense 
projects  are  demonstrating  new  technologies  to  provide  new 
capabilities  or  enhance  existing  capabilities,  there  are  many 
variables  to  juggle  such  as  cost,  schedule,  technology  maturity, 
requirements,  contracts,  and  staffing.  When  it  comes  to  deci¬ 
sions  that  Involve  assessing  several  random  variables  at  once, 
psychological  studies  have  shown  that  the  human  brain  has 
difficulty  thinking  forward  with  any  accuracy.  Moreover,  these 
papers  provide  evidence  that  the  most  simplistic  statistical 
models  are  more  accurate  than  human  predictions.  Based  on 
this  Information,  project  leaders  should  seek  and  use  data  to 
create  simple  charts  such  as  a  comparison  table,  histogram, 
Pareto  chart,  or  scatter  plot.  These  charts  will  enable  the  team 
to  view  and  analyze  data  and  perform  a  sensitivity  analysis  to 
understand  how  changing  one  variable  affects  the  other  vari¬ 
ables.  This  approach  will  allow  project  leaders  to  predict  future 
project  trends  and  understand  how  project  variables  Interact. 

The  cost  and  schedule  impacts  of  delaying  a  decision  can  be 
severe.  For  example,  a  complex  major  defense  acquisition 
program  may  have  2,000  full-time  equivalent  (FTE)  person¬ 
nel  employed  on  a  contract  including  government  personnel, 
prime  contractors,  subcontractors,  vendors,  and  suppliers. 
Assuming  a  yearly  cost  of  $400,000  per  FTE,  this  amounts  to 
an  approximate  cost  of  $16  million  per  week  on  the  contract. 
Now  if  work  is  stopped  for  2  weeks  while  a  decision  is  pend¬ 
ing  or  being  adjudicated,  the  impact  will  be  a  sunk  cost  of  $32 
million  and  a  schedule  slip  of  2  weeks  to  the  project. 

In  summary,  project  leaders  of  complex  DoD  projects  must 
make  timely  decisions  to  keep  the  progress  moving  forward 
and  to  maintain  high  motivation  levels.  Project  leaders  also 
should  utilize  data  and  implement  simple  quantitative  tech¬ 
niques  and  models  to  understand  sensitivities  and  interactions 
among  several  project  variables. 

Battle  Overzealous  Advocates 

Overzealous  advocates  are  overly  enthusiastic  individuals 
who  overpromise  and  underdeliver  on  projects.  While  proj¬ 
ect  advocates  can  have  a  positive  impact,  overzealous  advo¬ 
cates  promise  extraordinary  capabilities  at  a  fraction  of  the 
actual  cost  and  schedule.  These  advocates  can  be  extremely 
detrimental  to  the  long-term  prospects  of  a  complex  project 
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Figure  1.  Program  Cost  Overruns  from  Immature  Technology 
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since  they  develop  overly  optimistic  project  cost,  schedule, 
and  performance  baselines.  Overzealous  advocates  can  be 
senior  leaders  in  government  who  want  to  gain  positive  po¬ 
litical  light,  senior  leaders  in  private  industry  looking  to  win  a 
large  contract  that  will  produce  a  long-term  revenue  stream, 
and  government  program  managers  looking  to  be  promoted 
to  senior  government  or  military  ranks. 

Even  in  the  face  of  contrary  facts,  overzealous  advocates  will 
trend  to  optimistic  outcomes  Instead  of  realism.  Data  to  sup¬ 
port  this  viewpoint  are  presented  In  a  March  2008  article  I 
authored  on  best  project  management  and  system  engineer¬ 
ing  practices  for  large-scale  federal  acquisition  programs.  A 
few  comments  from  that  paper  include:  'The  program  suffered 
from  excess  optimism,"  "Frequent  turnover  makes  It  hard  to 
establish  accountability,"  "Decision  makers  need  to  reexamine 
decisions  as  new  Information  is  disclosed,"  and  "the  prime 
contractor  should  not  fear  retribution  for  bearing  bad  news." 
All  these  data  beg  the  question:  What  can  be  done  to  battle 
overzealous  advocacy?  Here  are  a  few  steps  that  may  help: 

■  Ensure  the  project  manager  and  key  team  members  are  as¬ 
signed  to  project  for  4  to  5  years  to  establish  accountability 
and  continuity  for  the  program  office. 

■  Conduct  an  unbiased,  independent  review  of  the  program 
with  outside  experts  prior  to  Milestone  B  and  at  key  design 
points  to  counter  overly  optimistic  estimates. 

■  Develop  a  detailed  end-to-end  risk  management  plan  in  the 
pre-acqusitlon  phase— prior  to  Milestone  B— that  Identifies 
program  risks  early  in  the  project  life  cycle.  This  is  crucial. 

■  Develop  rigorous  Milestone  B  entrance  and  exit  criteria 
and  ensure  they  are  adhered  to.  Issue  liens  if  the  criteria 
are  not  satisfied. 


In  summary,  the  best  methods  to  battle  overzealous  advocates 
on  DoD  projects  is  to  ensure  team  continuity  and  account¬ 
ability;  identify  and  document  all  risks  early  in  the  project  life 
cycle;  conduct  independent  review  prior  to  Milestone  B;  and 
develop  rigorous  entrance  and  exit  criteria  at  Milestone  B  and 
other  key  design  points. 

Mature  Technology  Early  and  in  a  Serial  Process 

Numerous  Government  Accountability  Office  (GAO)  reports 
detail  how  many  major  DoD  acquisition  programs  began 
the  program  execution  phase  without  verifying  that  critical 
project  technologies  had  reached  the  proper  maturity  level. 
As  shown  in  Figure  1,  data  collected  from  52  DoD  programs 
clearly  provide  evidence  that  not  maturing  technology  early  in 
the  program  life  cycle  had  a  factor  of  7  cost  growth  compared 
to  programs  that  had  matured  critical  technologies  at  the  ap¬ 
propriate  design  milestone. 

To  avoid  suffering  the  same  fate  as  many  of  the  programs  in 
Figure  1,  a  best  practice  is  to  mature  critical  technologies  to 
a  Technology  Readiness  Level  (TRL)  6  prior  to  Milestone  B. 
TRL  6  Is  defined  as  "testing  a  system  or  subsystem  model 
or  prototype  relevant  environment."  By  achieving  TRL  6,  the 
project  will  have  burned  down  significant  technology,  cost, 
and  schedule  risk. 

Another  best  practice  for  technology  in  complex  projects  is 
that  it  should  be  managed  in  a  serial  acquisition  process— 
not  in  parallel  with  system  development— in  order  to  lower 
risk  to  the  project.  For  example,  one  of  the  most  ambitious 
and  costly  programs  In  the  DoD,  the  Joint  Strike  Fighter  (JSF), 
implemented  a  concurrent  development  approach  and  has 
suffered  significant  cost  and  schedule  overruns. 
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Figure  2.  Prototypes  Enable  Earlier  Problem  Resolution 
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Instability  in  the  JSF  program  has 
been  and  continues  to  be  the  result 
of  highly  concurrent  development, 
testing,  and  production  activities. 

This  has  led  to  retrofitting  already 
procured  aircraft  to  correct  defi¬ 
ciencies  discovered  during  testing. 

The  JSF  is  a  complex  project  that 
is  trying  to  simultaneously  develop 
and  field  three  aircraft  variants  for 
the  Air  Force,  Navy,  Marine  Corps, 
and  eight  international  partners. 

With  respect  to  cost,  the  JSF  proj¬ 
ect  baseline  in  2001  was  for  2,866 
planes  at  a  total  acquisition  cost  of 
$233  billion  and  in  2012  skyrocketed 
to  a  total  cost  $395  billion  for  2,457 
planes.  Furthermore,  the  unit  cost 
per  aircraft  has  doubled  since  start 
of  development  in  2001  from  $69 
million  to  $137  million  in  2012. 

In  summary,  technology,  system,  and  testing  should  not  be 
done  concurrently.  A  good  rule  of  thumb  is  to  mature  technol¬ 
ogy  to  a  TRL  6  prior  to  Milestone  B.  By  meeting  this  technology 
metric,  a  project  will  burn  down  significant  project  risk  and 
reduce  the  likelihood  of  cost  overruns,  schedule  delays,  and 
meeting  technical  performance  requirements. 

Experiment  Early  and  Fail  Early 

Thomas  Edison  eloquently  captured  experimenting  early  and 
often  with  his  famous  quote,  "Negative  results  are  just  what  I 
want.  They're  just  as  valuable  to  me  as  positive  results.  I  can 
never  find  the  thing  that  does  the  job  best  until  I  find  the  ones 
that  don't."  Edison  was  well  aware  of  the  Importance  of  test¬ 
ing  early  through  rapid  and  frequent  experimentation.  He  was 
also  very  aware  that  failing  is  part  of  the  process  of  learning. 

Many  leading-edge  innovation  firms  exercise  Edison's  philoso¬ 
phy  by  testing  out  new  ideas  by  rapidly  building  mock-ups 
to  test  features  and  functions.  At  their  simplest  level,  mock- 
ups  may  take  the  form  of  cardboard,  clay,  papier-mache,  or 
three-dimensional  simulations.  The  idea  Is  to  quickly  build  a 
visual  representation  of  a  product  with  its  desired  functions 
and  features— a  prototype. 

Prototypes  can  serve  multiple  purposes.  They  can: 

■  Show  the  design  is  stable. 

■  Demonstrate  that  the  user  requirements  are  achievable. 

■  Serve  as  a  learning  tool. 

■  Provide  early  information  on  the  system. 

■  Encourage  communication  among  the  customer,  contrac¬ 
tor,  stakeholders,  and  team  members. 

■  Provide  a  planned  milestone  iteration  to  adjust  the  design 
specification. 

■  Serve  as  a  go/no-go  decision  point. 


Besides  serving  multiple  purposes,  prototypes  help  solve  pro¬ 
gram  issues  early  and  burn  down  risk  early  in  the  program  life 
cycle.  There  are  numerous  examples  in  the  literature— from 
automotive  climate  control  systems,  software  team  life  cycle 
approaches,  and  automotive  manufacturing— that  demon¬ 
strate  how  prototypes  significantly  reduce  manufacturing 
development  time  and  effort.  This  Is  particularly  relevant  for 
complex  DoD  defense  weapons  projects  that  manufacture 
large  quantities  of  weapons  systems. 

Figure  2  provides  a  graphical  description  of  how  building  pro¬ 
totypes  can  accelerate  problem  resolution  faster  compared 
to  traditional  developments  and  reduce  costly  rework,  which 
increases  by  roughly  a  factor  of  10  between  project  acquisi¬ 
tion  phases. 

In  summary,  build  prototypes— either  hardware  or  software— 
to  gain  knowledge  early,  to  reduce  technology  development 
time  and  effort,  and  to  solve  interface  issues  early.  This  ap¬ 
proach  will  enable  complex  DoD  projects  to  avoid  costly  re¬ 
work  and  subsequent  cost  overruns  and  schedule  delays  later 
in  the  project. 

Stop  Requirements  Creep 

Requirements  creep  Is  one  of  the  most  cited  reasons  for  cost 
overruns  and  schedule  delays  on  DoD  acquisition  projects. 
Stopping  requirements  creep  takes  exceptional  political  acu¬ 
men  and  a  deep  understanding  of  systematic  impacts.  When 
pressured  to  change  requirements.  It  Is  the  project  leader's 
job  to  explain  to  stakeholders  that  changing  requirements  in 
the  project  execution  phase  usually  leads  to  program  cost  and 
schedule  overruns. 

In  order  to  support  this  view,  let's  look  at  GAO  data  in  Figure 
3  that  show  the  impacts  of  changing  requirements.  Figure 
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Figure  3.  Program  Impacts  of  Changing  Requirements 
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Average  cost  growth  for  programs  over  initial  estimates 
(GAO-09-326SP)  from  data  on  52  programs. 


Average  delay  in  providing  initial  operating  capability 
(GAO-09-326SP)  from  data  on  52  programs. 


3  (left)  provides  data  on  52  DoD  weapons  programs  that 
changed  requirements  and  shows  that  these  programs  suf¬ 
fered  average  cost  growths  of  greater  than  a  factor  of  3,  and 
Figure  3  (right)  shows  that  the  average  schedule  delay  Is 
greater  than  a  factor  of  2,  compared  to  programs  that  did 
not  change  requirements. 

In  general,  requirements  change  for  several  reasons:  too  many 
stakeholders  with  divergent  needs  and  wants;  no  project  ap¬ 
proved  requirements  baseline  at  Milestone  B;  and  agencies 
routinely  accepting  requirements  changes  post-Milestone  B 
with  no  understanding  of  system  impacts. 

Requirement  changes  are  a  widespread  problem  in  the  DoD, 
and  strong  leadership  Is  required  to  combat  this  trend.  The 
most  effective  approach  to  avoid  requirement  changes  is  to 
enact  the  following  steps: 

■  Have  a  vetted,  approved  requirements  baseline  prior  to 
Milestone  B. 

■  Implement  a  no-change  requirements  policy.  Stick  to  it. 

■  Implement  a  change  control  board  (CCB)  and  mandate  a 
cost-benefit  evaluation  for  any  requirement  change. 

■  Have  a  strong,  politically  astute  project  champion  to  help 
manage  stakeholders. 

Minimizing  or  having  no  requirements  changes  gives  com¬ 
plex  projects  a  chance  to  deliver  a  system  that  meets  cost, 
schedule,  and  technical  targets.  In  summary,  have  a  vetted  set 
of  requirements  early  In  the  project;  have  a  government-led 
CCB;  and,  most  important,  have  a  strong  project  champion. 


Take  Great  Care  Managing  Interfaces 

Many  complex  projects  suffer  setbacks  and  failures  by  not 
clearly  defining  technical  and  organizational  interfaces.  In¬ 
terfaces  are  points  where  a  transfer  of  Information  occurs.  A 
technical  interface  can  be  an  optical,  mechanical,  electrical, 
thermal,  or  data  transfer  point.  Internal  organizational  inter¬ 
faces  can  be  among  the  project  managers,  system  engineers, 
contracts  leads,  budget  leads,  designers,  builder,  testers,  oper¬ 
ators,  and  users,  while  external  organizational  interfaces  may 
occur  between  government  agencies,  the  prime  contractor, 
and  subcontractor.  As  one  can  imagine,  a  complex  DoD  proj¬ 
ect  may  have  millions  of  technical  interfaces  and  tens  of  orga¬ 
nizational  interfaces  that  need  to  be  managed  (see  Figure  4). 

On  most  programs,  the  technical  interfaces  are  managed  by 
a  system  engineering  integrated  product  team  (IPT).  This  IPT 
Is  tasked  with  ensuring  all  interfaces  are  captured,  specified 
accurately,  and  documented  when  changes  occur.  One  best 
practice  is  to  create  a  comprehensive,  detailed  interface  con¬ 
trol  document  (ICD)  that  identifies  and  documents  all  project 
Interfaces  as  well  as  all  unattended  and  mismatched  inter¬ 
faces.  The  ICD  also  should  contain  a  configuration  manage¬ 
ment  (CM)  plan  to  document  and  communicate  all  interface 
changes  to  the  project  team.  There  also  should  be  a  change 
control  board  (CCB)  that  meets  daily  or  weekly  to  discuss  and 
communicate  interface  changes.  The  organizational  interfaces 
should  be  captured  in  a  stakeholder  communication  plan. 

Another  best  practice  to  minimize  interface  control  and  plan 
for  obsolescence  is  to  design  In  modularity  and  commonal¬ 
ity  to  the  system  under  development.  Modularity  refers  to 
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Figure  4.  Technical  and  Organizational 
Interface  Management 


Manage  the 
Interfaces 


should  ensure  that  most  the  efficient  software  develop¬ 
ment  approach,  such  as  spiral,  agile,  or  waterfall  be  uti¬ 
lized.  This  task  should  be  led  by  a  software  integrated 
product  team  (IPT). 

Another  best  practice  for  software  development  Is  to  per¬ 
form  rigorous  regression  testing  for  any  software  change. 
On  one  highly  successful,  large-scale,  complex  software 
project  for  a  ground  station,  the  contractor  team  imple¬ 
mented  regression  testing  on  every  new  or  modified  line 
of  code  and  delivered  the  software  system  to  the  ground 
site  with  zero  errors.  Finally,  prior  to  developing  software, 
the  project  manager  should  take  into  account  the  final 
system  configuration  and  ensure  that  the  development 
code  and  software  system  interfaces  are  compatible,  the 
computational  complexity  Is  not  too  high,  and  that  the  al¬ 
gorithms  meet  the  system  specifications  and  are  scalable. 


designing  In  volumes  on  a  system  that  can  accommodate 
future  technology  2,  4,  or  8  times  larger  or  smaller,  while 
commonality  refers  to  using  standard  interfaces.  Additional 
benefits  of  incorporating  modularity  and  commonality  are 
that  they  will  reduce  the  number  of  interfaces  to  manage  and 
reduce  switching  costs  for  future  systems. 


In  summary,  treat  a  DoD  software  project  like  a  hardware 
project  with  phases  and  milestones;  create  a  software 
IPT;  utilize  a  development  lifecycle  consistent  with  the 
project's  complexity  and  requirements;  track  and  docu¬ 
ment  software  interfaces;  ensure  the  software  Is  scalable; 
and,  finally,  perform  regression  testing  to  ensure  that  a  high- 
quality  product  that  meets  all  specifications  Is  delivered  to 
the  final  system. 

Summary 


By  creating  a  comprehensive  ICD,  documenting  and  com¬ 
municating  interface  changes  with  a  rigorous  CM  process, 
creating  a  stakeholder  communication  plan,  and  incorporat¬ 
ing  in  modularity  and  commonality,  a  project  manager  will 
significantly  decrease  the  likelihood  of  interface  Issues  on  a 
complex  project. 

Create  a  Software  Integrated  Product  Team 

Functions  performed  by  software  continue  to  increase  on 
many  DoD  weapons  systems.  For  instance,  data  from  the  2010 
House  Armed  Services  Committee  (HASC)  report  show  that 
the  percent  of  functions  performed  by  software  has  Increased 
considerably  over  the  past  few  decades  on  several  weapons 
systems  (see  Table  1). 


There  is  a  quote  from  Albert  Einstein  that  is  very  relevant  to 
complex  projects:  "Any  intelligent  fool  can  make  things  bigger 
and  more  complex. ...  It  takes  a  touch  of  genius— and  a  lot  of 
courage— to  move  in  the  opposite  direction."  In  many  cases,  it 
Is  organizations,  agencies,  and  senior  committees  that  make 
projects  bigger  and  more  complex  than  they  need  to  be.  My 
hope  is  that  this  article  will  provide  leaders  of  complex  projects 
with  the  data  and  the  courage  to  reduce  complexity  and  deliver 
complex  projects  within  scope,  cost,  and  schedule. 

The  author  can  be  contacted  at  srmeier@srmconsultmgllc.com. 


Table  1.  Percentage  of  Functions 
Performed  by  Software  on  Several 
Weapons  Systems 


The  same  report  provides  dismal  statistics  on  the  success 
rate  of  DoD  IT  projects:  Only  16  percent  of  IT  projects  were 
completed  on  time  and  on  budget,  31  percent  were  canceled 
before  completion,  and  53  percent  were  late  or  over  bud¬ 
get  with  typical  cost  growths  exceeding  89  percent.  Even 
more  disturbing  Is  that  of  the  IT  projects  completed,  the  final 
products  contained  only  61  percent  of  the  originally  speci¬ 
fied  features.  This  is  a  poor  report  card  for  DoD  software 
development  programs. 

The  leader  of  a  complex  DoD  project  should  ensure  that 
software  development  be  treated  the  same  as  hardware, 
with  phases  and  milestones.  In  addition,  the  project  manager 
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The  Need  for 
Agile  Program 
Documentation 


LTC  TJ.  Wright,  USA 


The  past  decade  has  brought  much 
change  for  warfighters,  necessitating 
new  materiel  solutions  to  ensure  our  soldiers 
have  what  they  need  to  accomplish  the  mission. 

Over  the  course  of  the  last  10  years,  doctrine,  strat¬ 
egy,  operations,  tactics,  techniques,  procedures,  as  well  as  the 
threat  and  battlefield  environments  have  changed  significantly. 


From  the  initial  invasion,  to  the  withdrawal  from  Iraq,  to  the  transfer  of  mission  to  Afghanistan,  requirements  for 
soldier's  equipment,  vehicles,  surveillance,  and  weapons  systems  have  challenged  materiel  developers  to  keep  pace 
with  the  speed  of  war.  A  known  program  that  demonstrated  the  government's  remarkable  ability  to  streamline  the 
process  to  develop,  evaluate,  and  field  within  2  years  Is  the  Mine  Resistant  Ambush  Protected  Vehicle.  When  an 
urgent  requirement  necessitated  a  rapid  response,  all  the  stakeholders  from  the  resourcers,  developers,  evaluators, 
and  sustainers  executed  a  more  streamlined  process  to  get  capability  to  the  field  faster,  albeit  with  some  challenges. 


Wright  is  the  US.  Army  product  manager  for  Precision  Guided  Missiles  and  Rockets,  an  AC  AT  1C  program.  He  holds  a  B.S.  degree  from  the 
United  States  Military  Academy,  an  M.E.  from  the  University  of  Virginia,  and  he  is  a  graduate  of  the  Army  Command  and  General  Staff  College. 
He  is  a  member  of  the  Army  Acguisition  Corps  with  a  Level  III  certification  in  program  management  and  is  an  Army-certified  Lean  Six  Sigma 
(LSS)  Black  Belt. 
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Table  1.  Milestone  B  Documentation 


OSD  Statutory 


OSD  Regulatory 


Army  Regulatory 


Acquisition  Program  Baseline  (APB) 
Analysis  of  Alternatives  (AoA) 
Benefit  Analysis  and  Determination 
Business  Case  Analysis  (for  2366b) 
Clinger-Cohen  Act  (CCA)  Compli¬ 
ance 

Competition  Analysis 
Consideration  of  Technology  Issues 
Cooperative  Opportunities 
Core  Logistics  Analysis/Source  of 
Repair 

Data  Management  Strategy 
Determination  of  Contract  Type 
Independent  Cost  Estimate  (ICE) 
Industrial  Base  Capabilities  Consid¬ 
erations 

Low  Rate  Initial  Production  (LRIP) 

Quantities 

Manpower  Estimate 

Market  Research 

MDA  Program  Certification 

Post  Implementation  Review 

PESHE 

Replaced  System  Sustainment  Plan 
Selected  Acquisition  Report  (SAR) 
Submission  of  DD  Form  1492  &  Cert, 
of  Spectrum  Support 


Acquisition  Decision  Memorandum 
(ADM) 

Acquisition  lA  Strategy 

Acquisition  Strategy 

Affordability  Assessment 

Capability  Development  Document  (CDD) 

CIO  Confirmation  of  CCA  Compliance 

Corrosion  Prevention  Control  Plan 

CARD 

DoD  Component  Cost  Estimate 
Exit  Criteria 

Information  Support  Plan  (ISP) 

Initial  Capabilities  Document  (ICD) 

Item  Unique  Identification  (lUID)  Plan 
Life  Cycle  Signature  Support  Plan 
Life  Cycle  Support  Plan  (LCSP) 

MDA  Assess  of  compliance  w/CBRN  Rqmt 
Net-Centric  Data  Strategy 
OTA  Report  of  OT&E  Results 
Preliminary  Design  Review  (PDR)  Report 
PPP  for  Programs  with  CPI 
Spectrum  Supportability  Determination 
Staffing  Plan 

System  Security  Management  Plan 
System  Threat  Assessment  Report  (STAR) 
Systems  Engineering  Plan  (SEP) 

Technology  Readiness  Assessment 
Test  and  Evaluation  Master  Plan  (TEMP) 


Acquisition  Plan 

Applied  Embedded  Diagnostic  Assess¬ 
ment  Memo 

Army  Cost  Position  (ACP) 

Basis  of  Issue  Plan  (BOIP)/Qualitative 
and  Quantitative  Personnel  Reqmts. 
Info  (QQPRI) 

Business  Case  Analysis 
CPI  Identification  Memo 
Environmental  Quality  Life-Cycle  Cost 
Estimate 

Interoperability  Certification— I ntra 
Army 

MANPRINT  Assessment/MER 
Materiel  Fielding  Plan  draft 
MIPS 

New  Equipment  Training  Plan  (NETP) 
Performance-Based  Agreement 
Safety  Release  (if  req'd) 

Safety  Confirmation 
Simulation  Support  Plan  (SSP) 

System  Safety  Management  Plan 
System  Training  Plan  (STRAP) 
Transportability  Report/Transportabil¬ 
ity  Assessment 
Program  Office  Estimate 


Is  It  possible  to  place  this  concept  across  the  Army  and  De¬ 
partment  of  Defense  (DoD)? 

The  Challenge 

In  most  programs,  the  lack  of  an  urgent  requirement  dictates 
the  standard  acquisition  process  with  its  historical  use  of 
lengthy  and  costly  program  resources.  In  2010,  Secretary  of 
the  Army  John  McHugh  stated  in  an  Acquisition  Review  that, 
"'We  need  an  agile  system  that  rapidly  develops,  purchases, 
and  fields  innovative  solutions  for  our  soldiers."  Recently,  se¬ 
nior  defense  leaders  have  directed  program  managers  to  pur¬ 
sue  avenues  that  are  smarter  and  more  efficient  and  to  pursue 
optimal  program  structures  to  deliver  capability  that  aren't  just 
cookie  cutter  program  plans.  However,  the  modernization  of 
the  current  documentation  requirements  has  not  kept  pace 
with  this  optimal  guidance  and  does  not  readily  support  non- 
traditional  approaches.  There  Is  a  critical  need  for  the  defense 
acquisition  community  to  create  a  more  agile  documentation 
process  to  support  and  permit  the  documented  approval  of 
programs  that  will  rapidly  and  timely  provide  the  warfighter 
with  the  capability  to  defeat  current  and  potential  adversaries 
in  future  contingencies. 

The  Current  Documentation  Process 

The  common  denominator  for  coordinating  a  program  across 
the  required  DoD  offices  and  agencies  Is  documentation. 


Yet,  the  traditional  documentation  requirements  are  a  com¬ 
mon  factor  of  extended  program  schedule.  There  are  ap¬ 
proximately  70  statutory  and  regulatory  documents  required 
to  successfully  negotiate  a  major  program  milestone.  Each 
document  necessitates  considerable  man-hours  to  write,  co¬ 
ordinate  within  the  program  office,  and  staff  across  dozens  of 
higher  echelon  offices;  the  program  executive  officer.  Army, 
and  DoD.  Additionally,  rework  and  rewriting  due  to  frequent 
changes  to  templates  or  documentation  increase  the  already 
significant  resources  spent  from  start  to  final  approval. 

Significant  resources  are  spent  developing,  coordinating,  and 
staffing  the  program  support  documentation  for  a  Materiel 
Development  Decision,  Milestones  A,  B,  C,  and  the  Full  Rate 
Production  decision.  This  environment  limits  the  acquisition 
process  responsiveness.  By  the  time  a  weapon  system  Is 
fielded  (as  long  as  7  years  per  the  DoDI  5000  series),  the 
doctrine,  strategy,  and  theater  may  have  changed  and  the 
tactics,  techniques,  and  procedures  may  even  require  a  new 
materiel  solution.  We  need  to  review  the  required  documen¬ 
tation  to  reflect  the  improvement  we  are  witnessing  In  rapidly 
developing  and  fielding  program  capabilities. 

Streamline  Required  Documentation 

Streamlining  the  documentation  process  can  be  accom¬ 
plished  simply  by  more  extensively  tailoring  required 
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documentation  to  the  program,  eliminating  nonvalue-added 
documentation,  and  postponing  the  submission  of  low-risk 
documents  until  after  fielding.  Maybe  there  is  even  a  quick- 
look  review  for  some  documentation  requirements  to  be  fol¬ 
lowed  by  a  more  extensive  review  and  submission  for  specific 
low-risk  cases.  These  cases  could  be  identified  by  higher 
technology  readiness  levels,  established  production  capa¬ 
bility,  or  commonality  of  system  reuse  due  to  an  incremental 
upgrade.  The  goal  is  to  deliver  a  safe  and  reliable  product  to 
the  warfighter  as  quickly  as  possible  and  in  some  cases  fol¬ 
low  up  with  the  required  documentation  where  technology 
maturity  allows. 


review  and  approval.  Workforce  members  need  to  be  em¬ 
powered  to  get  rid  of  the  status  quo  and,  more  important, 
allow  nontraditional  approaches.  There  are  numerous  lean 
methods  to  revamp  the  documentation  process  and  only 
require  documents  where  there  is  value  added  in  delivering 
capability  effectively  and  efficiently  to  the  warfighter:  value 
stream  mapping,  cutting  redundancies  and  process  delays, 
and  minimizing  unnecessary  reviews  through  internal  and 
external  agencies. 

Another  method  is  to  delegate  authority  for  approving 
documentation  to  the  lowest  level  possible  and  ensure 


When  an  urgent  requirement  necessitated  a 
rapid  response,  all  the  stakeholders  from  the 
resourcers,  developers,  evaluators,  and 
sustainers  executed  a  more 
streamlined  process  to  get  capability 
to  the  field  faster,  albeit  with 
some  challenges. 


Let  Program  Purpose  Drive  Documentation 

The  primary  purpose  of  a  tactical  commander's  intent  is  to 
provide  the  framework  for  subordinate  actions.  Why  does 
the  documentation  not  follow  the  same  concept?  The  more 
the  program  manager's  freedom  of  movement  Is  limited,  the 
fewer  are  the  means  and  methods  available  to  pursue  an  op¬ 
timal  program  structure.  Rather  than  document  proponents 
reviewing  document  responses  to  ensure  they  satisfy  "go/ 
no-go  lists,"  a  better  process  might  entail  reviewing  responses 
to  ensure  they  meet  the  document's  intent  at  an  acceptable 
risk  level.  In  some  cases,  this  could  save  substantial  time 
by  focusing  on  what  is  truly  needed  to  assess  the  risk  while 
minimizing  nonvalue-added  time  and  effort.  If  a  document's 
content  meets  the  intent  with  little  risk,  it  is  sufficient.  How 
can  we  emplace  a  program  that  requires  only  the  necessary 
documents,  assesses  program  risk,  and  is  approved  once  the 
intent  of  the  document  is  met? 

Revise  to  Reduce  Review  and  Approval  Steps 

The  key  to  affecting  a  documentation  paradigm  shift  Is  a 
collective  enterprise  response  to  changing  the  way  we  do 
business.  The  defense  acquisition  community  Is  receiving 
well  strategized  and  insightful  guidance  from  our  senior 
defense  leaders.  The  challenge  is  the  implementation  of  a 
process  that  supports  that  guidance.  This  change  will  not 
be  easy,  especially  for  organizations  that  have  a  substantial 
number  of  personnel  assigned  the  task  of  documentation 


accountability  while  enforcing  a  new  process,  incentivizing 
creativity,  and  rewarding  efficiency. 

A  concerted  effort  is  necessary  to  align  our  warfighters' 
needs  to  defeat  current  and  potential  adversaries  in  future 
contingencies  with  our  obligations  to  the  taxpayer.  To  effect 
a  significant  paradigm  shift,  leadership  at  each  level  must 
support  process  change. 

One  good  candidate  Is  the  reduction  of  the  number  of  required 
supporting  documents  and  the  process  used  to  staff  and  ap¬ 
prove  them.  Navigating  the  existing  documentation  process 
in  pursuit  of  the  optimal  structure  will  continue  to  be  difficult 
unless  the  "document  checklist  mentality  process/method" 
Is  changed  to  a  more  purpose-driven  process  that  focuses 
only  on  what  Is  necessary  to  deliver  capability  to  the  field  as 
efficiently  as  possible.  Program  managers  should  be  granted 
authority  to  meet  the  "intent"  of  only  publishing  those  docu¬ 
ments  that  apply  to  their  programs  within  an  acceptable  level 
of  risk.  The  current  acquisition  system  cannot  accept  this  rec¬ 
ommended  "program-specific  purpose  driven  documentation" 
paradigm  without  senior  leadership  support,  and  likewise  the 
document  owners  embracing  process  change  in  how  we  co¬ 
ordinate  documentation  with  the  program  stakeholders. 


The  author  can  be  contacted  at  todd.j.wnght@us.army.mn. 
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cquisition  community  members  are 
pWi  of  a  team  tasked  with  making 
affordable  and  operationally  effec¬ 
tive  Qrocurement  decisions  for  the 
Oepal\tment  of  Defense  (DoD).  To 
achieve  this  goal,  workforce  engi- 
neersand  engineering  teams  must 
have  maintain  a  well-balanced 
skill  set  that  includes  an  under¬ 
standing  of  government  acquisi¬ 
tion  policies  and  technical  skills 
that  provide  the  level  of  expertise 
required  for  their  role  in  the  acqui¬ 
sition  process. 

Providing  acquisition  workforce  engineers  this  skill  set  balance  requires 
a  partnership  between  the  acquisition  and  technical  communities  within 
DoD.  The  Defense  Acquisition  University  (DAU)  has  taken  on  the  role  of 
providing  acquisition  workers  the  skill  sets  required  for  success  in  learning 
the  required  acquisitions  policies  and  procedures  for  various  acquisition 
roles.  The  training  provided  is  directly  applicable,  progressive,  career- 
long,  and  relevant  to  a  particular  DoD  department.  On  the  other  hand,  the 


Galway,  a  professional  engineer,  is  a  mechanical  engineer  for  the  U.S.  Navy's  Combatant 
Craft  Division  C832-Systems  Design  and  Integration  Branch  in  Norfolk,  Va.  He  is  DAW  I A 
Level  III  certified  in  Systems  Planning,  Research,  Development,  and  Engineering  and  he  has 
more  than  27  years  of  experience  in  both  government  and  private  industry. 
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applicable  technical  skill  set  is  not  being  maintained  with  as 
much  structure,  consistency,  or  resourcing. 

Of  particular  concern  to  the  acquisition  community  should 
be  development  of  the  technical  skill  sets  needed  to  support 
complex  roles  requiring  multisystem  technical  requirement 
apportioning,  balancing  system  life-cycle  needs  during  acqui¬ 
sition  phases,  and  providing  the  capability  to  create  affordable 
design  engineering  solutions  to  problems.  Developing  these 
specific  technical  skill  sets  involves  lengthy  and  specific  de¬ 
velopmental  experiences  for  government  engineering  per¬ 
sonnel.  Acquiring  the  necessary  skills  through  random  work 
experiences  alone  may  take  a  substantial  portion  of  a  typical 
government  engineering  career.  The  barriers  to  developing 
these  skill  sets  include  outsourcing  engineering  work,  resourc¬ 
ing  long-term  progressive  training  programs,  lack  of  technical 
knowledge  management,  career  transitions,  and  many  other 
factors.  The  purpose  of  this  effort  Is  to  identify  some  of  the 
Issues  related  to  the  technical  side  of  this  partnership  and  sug¬ 
gest  a  strategy  for  improving  the  engineering  skill  sets  most 
relevant  to  supporting  the  acquisition  community. 

Engineers  typically  come  into  government  service  with  a  degree 
in  a  very  general  field  of  engineering  (electrical,  mechanical, 
civil,  etc.).  Upon  entry  into  government  service,  they  begin  to 
learn  how  to  apply  these  general  engineering  skills  to  the  spe¬ 
cific  needs  of  their  new  employer.  During  the  Initial  indoctrina¬ 
tion  period,  there  typically  is  either  a  formal  or  Informal  intern¬ 
ship  where  new  engineers  learn  the  processes,  practices,  and 
procedures  of  their  new  jobs.  In  this  same  period,  they  start  to 


management  or  engineering  management  track.  The  missing 
track  from  this  list  is  a  track  providing  structured  long-term 
technical  developmental  programs  for  complex  generalized 
roles,  such  as  design  engineer  and  systems  engineer.  These 
roles  develop  service-specific  innovation  and  production  heu¬ 
ristics  that  are  the  source  for  the  sound  engineering  judgments 
and  creative  intuition  for  resolving  acquisition  program  engi¬ 
neering  issues.  Collectively,  personnel  engaged  in  these  roles 
are  the  backbone  for  DoD  technical  core  competency. 

The  most  essential  element  for  engineers  on  a  complex  gener¬ 
alized  track  is  the  need  to  actually  do  the  technical  work  under 
the  supervision  of  an  experienced  engineering  mentor.  Like 
similar  programs,  substantial  mentor  Involvement  is  needed 
initially,  followed  by  a  transitional  period  where  mentoring  is 
reduced  and  independent  work  becomes  only  occasionally  re¬ 
viewed.  Gradually,  the  mentor  becomes  more  of  a  colleague  or 
consultant  on  a  multilevel  engineering  team.  A  certain  amount 
of  actual  core  competency  work  also  must  be  accomplished 
throughout  a  career  just  to  stay  in  practice  and  capable  of  inte¬ 
grating  new  materials,  technology,  and  systems  into  projects. 
For  larger  and  more  complex  projects,  you  need  to  be  able 
to  readily  immerse  yourself  in  the  technical  design  without 
spending  too  much  time  getting  up  to  speed  with  the  latest 
technological  advances.  Practice  is  in  contrast  to  being  the 
government  technical  point  of  contact  (TPOC)  controlling  the 
work,  where  the  engineer  is  the  person  responsible  for  techni¬ 
cal  oversight  of  a  contractor's  work.  This  is  not  to  say  control¬ 
ling  work  should  not  also  be  part  of  the  learning  experience, 
but  It  Is  to  say  that  enough  work  needs  to  be  accomplished 


Engineering  managers  face  balancing  the  challenges  and  technical 
problems  of  paying  customers  with  training  the  workforce  in 
a  “working  capital  funding"  environment.  Often,  training  must 
take  a  back  seat  to  product  delivery. 


become  aware  of  their  customer's  needs,  available  resources, 
and  working  both  as  an  individual  and  team  member  in  projects. 
This  period  may  last  a  year  or  two,  it  Is  very  command-unique, 
and  it  Is  not  the  time  of  primary  concern  in  this  effort. 

After  the  initial  indoctrination,  most  engineers  start  to  develop 
in  what  might  be  considered  a  mentored  developmental  train¬ 
ing  period,  perhaps  analogous  to  a  medical  residency.  This  will 
involve  on-the-job  training,  completion  of  Increasingly  more 
complex  assignments,  and  learning  how  to  function  indepen¬ 
dently  as  an  engineer.  Some  will  enter  into  specific  government 
training  programs,  such  a  those  under  the  Defense  Acquisition 
Workforce  Improvement  Act  (DAWIA).  Some  will  just  begin 
work  as  journeyman-level  engineers.  Some  will  go  on  to  addi¬ 
tional  education  with  graduate  academic  work  as  they  go  down 
the  technical  specialist  track.  Others  will  go  down  a  project 


by  the  engineer  to  achieve  initial  proficiency  In  the  role  and 
then  maintain  proficiency  in  the  role  throughout  their  careers. 

As  simple  as  this  sounds,  it  becomes  increasingly  more  dif¬ 
ficult  to  get  relevant  and  challenging  engineering  assignments 
that  enable  staying  in  practice  as  you  become  a  more  senior 
engineer,  largely  due  to  role  shifts  caused  by  the  acquisition 
reform  of  the  1990s.  In  addition  to  these  shifts,  work  that  is 
difficult  to  contract  out  resulting  from  unusual  circumstances, 
such  as  extreme  schedule  constraints,  politically  charged  is¬ 
sues,  or  even  availability  of  contracts,  all  tend  to  supersede 
the  need  for  government  engineers  to  work  on  core  technical 
work.  Reducing  the  opportunity  further  is  the  perception  that 
contracting  out  such  work  is  a  cheaper  way  to  accomplish  a 
task  and  that  one  engineer  can  oversee  much  more  than  a 
single  person  can  do  alone.  A  working  capital-funded  program 
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is  reluctant  to  assume  any  of  the  financial  burden  associated 
with  maintaining  technical  core  competency  of  engineering 
workers.  The  long-term  effect  of  engineers  not  engaging  in 
technically  challenging  work  also  is  not  captured  by  short-term 
price  comparisons.  Not  accounting  for  this  long-term  resource 
loss  leads  to  a  diminished  and  dated  command  collective  tech¬ 
nical  resource  capability.  The  degradation  is  difficult  to  mea¬ 
sure  and  often  masked  by  inflated  technical-sounding  titles 
given  to  work  assignments  that  are  in  reality  more  administra¬ 
tive  than  technical.  There  also  is  an  employee-driven  general 
shift  from  technical  engineering  to  project  engineering  and 
engineering  administration  because  it  usually  is  the  path  to 
greater  compensation  for  time. 


For  example,  one  of  the  roles  that  requires  a  long  develop¬ 
ment  period  and  constant  practice  for  proficiency  Is  that 
of  design  engineer.  Design  engineers  are  the  creators  of 
the  artifacts  used  to  realize  how  mission  requirements 
can  be  met  in  a  safe  and  suitable  manner.  They  are  the 
front-line  workers  in  technical  risk  decisions,  integration  of 
concepts,  and  determining  a  reasonable  tradeoff  strategy 
in  production  efforts.  New  engineers  taking  on  the  role  of 
design  engineer  must  find  creative  and  affordable  solutions 
to  meet  mission  requirements  using  academic  principles, 
industry  products,  and  production  practices.  This  involves 
a  constant  iterative  comparison  between  product  costs, 
most  effective  production  process,  material  constraints. 


If  we  want  engineers  to  stay  in  these  complex  general  role 
tracks,  a  structured  development  plan  is  needed  for  quantifying 

and  achieving  the  expertise. 


If  we  want  engineers  to  stay  in  these  complex  general 
role  tracks,  a  structured  development  plan  is  needed  for 
quantifying  and  achieving  the  expertise,  matched  by  a 
compensation  plan  that  equates  their  importance  to  the 
acquisition  program. 

Senior  engineers  traditionally  have  been  Informally  charged 
with  mentoring  the  next  generation,  communicating  the 
knowledge  associated  with  specific  past  experiences,  and 
providing  life-cycle  engineering  support  for  past  and  present 
acquisitions.  The  new  trend  appears  to  be  project  engineering, 
where  the  oversight  of  many  contracts  or  projects  amplifies 
the  influence  of  an  engineer.  However,  such  a  work  strategy 
precludes  engineers  from  having  the  time  to  accomplish  com¬ 
plex  engineering  developmental  assignments  that  demand 
continuity  of  thought  and  focus  on  a  specific  complex  set  of 
Issues.  A  sad  byproduct  of  this  strategy  also  is  a  diminished  ca¬ 
pacity  to  mentor.  Loss  of  the  opportunity  to  complete  complex 
technical  core  competency  engineering  assignments  equates 
to  reduced  engineering  proficiency. 

A  loss  in  opportunity  to  transfer  knowledge  or  mentor  young 
engineers  is  a  lost  training  opportunity.  Engineering  managers 
face  balancing  the  challenges  and  technical  problems  of  paying 
customers  with  training  the  workforce  in  a  "working  capital 
funding"  environment.  Often,  training  must  take  a  back  seat 
to  product  delivery.  This  creates  a  learning  environment  that 
is  often  sporadic,  inconsistent,  and  fragmented.  Engineering 
roles  requiring  long  developmental  training  periods  are  par¬ 
ticularly  hurt  by  this  type  of  learning  environment.  A  struc¬ 
tured  development  program  for  these  roles  would  assist  in 
managing  these  resources.  A  technical  version  of  what  DAU 
provides  DAWIA  workers  would  provide  a  means  to  manage 
the  training  of  engineering  resources  to  support  the  complex 
roles  associated  with  large  acquisition  programs. 


safety  and  environmental  regulations,  and  many  other 
factors.  These  solutions  must  be  technically  sound,  com¬ 
municated  to  the  production  workforce,  tested,  logistically 
supported,  and  properly  archived.  The  time  invested  in  this 
role  includes  learning  and  staying  abreast  of  industry  prod¬ 
ucts,  production  techniques,  performance  of  equipment 
in  the  field,  and  production  costs.  Most  new  designs  also 
include  the  challenge  of  integrating  them  into  the  existing 
systems  and  operational  procedures.  Effective  integration 
of  new  designs  into  existing  products  and  systems  Is  a  skill 
that  takes  practice  to  learn.  However,  the  dividends  from 
this  time  investment  Include  increased  vision  about  the 
probability  of  success  of  new  concepts,  and  understand¬ 
ing  about  the  dominant  design  factors,  knowledge  of  the 
controlling  cost  factors,  and  an  ability  to  rapidly  identify  the 
impact  of  changes  to  operational  or  design  requirements. 
These  attributes  are  important  technical  support  skills  to 
be  able  to  bring  to  an  acquisition  program.  As  a  side  note, 
acquisition  reform  and  the  trend  to  contract  out  the  design 
engineering  function  have  reduced  the  opportunities  for  de¬ 
sign  engineering  development  programs,  particularly  within 
the  subset  of  acquisition  workforce  members. 

A  second  role  that  requires  a  long  development  period  is 
that  of  systems  engineer  for  complex  systems.  The  techni¬ 
cal  side  of  systems  engineering  involves  at  least  a  functional 
understanding  of  how  systems  work,  how  they  interact  with 
the  environment,  and  how  they  interact  with  other  systems. 
In  the  case  of  complex  equipment,  systems  engineers  need 
to  understand  the  balance  between  individual  system  per¬ 
formance  and  the  overarching  performance  of  the  total  mis¬ 
sion  system.  For  example,  typically  desirable  skill  sets  include 
understanding  Issues  such  as  apportionment  of  power  re¬ 
sources  or  weight  allowance  for  different  systems  to  optimize 
total  performance  of  a  vehicle. 
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Keeping  abreast  of  the  various  systems,  given  the  rate  of 
change  in  many  industries,  can  be  a  full-time  job.  However, 
systems  engineers  also  need  to  know  and  understand  the  ac¬ 
quisition  process  and  understand  how  to  work  through  issues 
associated  with  the  different  steps  in  the  process.  Because 
each  acquisition  is  different,  this  often  involves  learning  how 
to  apply  and  adapt  procedures  to  situations  at  hand  in  addi¬ 
tion  to  knowing  the  defined  procedures.  Frequently,  systems 
engineers  start  in  one  discipline  (such  as  mechanical,  electri¬ 
cal,  or  structural),  then  learn  how  systems  in  their  field  inter¬ 
act  with  other  systems  in  complex  equipment.  Consequently, 
in  addition  to  keeping  current  with  systems  in  their  field  and 
acquisition  procedures,  considerable  time  is  spent  learning 
and  understanding  the  changes  in  system  interaction  as  a 
result  of  changes  to  other  systems.  Systems  engineers  often 
can  find  theirtime  constrained  by  involvement  in  many  paral¬ 
lel  projects,  often  at  different  phases  of  an  acquisition,  and 
must  keep  up  with  changes  to  acquisition  procedures  at  all 
phases.  For  their  investment  of  time  in  learning  the  breadth  of 


forward.  There  is  enough  commonality  of  information  in 
both  roles  for  there  to  be  substantial  benefit  in  an  "on¬ 
line"  technical  knowledge  management  system  for  both 
roles  within  the  government.  Such  a  system  would  not  only 
capture  the  information,  but  allow  it  to  be  maintained  and 
monitored  in  a  manner  consistent  with  the  individual  tech¬ 
nical  authorities  within  DoD.  Ideally,  a  technical  knowl¬ 
edge  management  system  also  would  permit  capturing  the 
"lessons  learned"  by  the  workforce  as  well  as  delivering 
the  policies  of  technical  authorities. 

The  DoD  acquisition  process  is  designed  to  provide  a  deli¬ 
cate  balance  between  flexibility  and  risk  that  needs  an  effec¬ 
tive  technical  leg  with  awareness  of  acquisition  policies  and 
products.  Creation  and  implementation  of  these  products  by 
the  acquisition  workforce  in  an  affordable  and  operationally 
effective  manner  depends  on  the  existence  and  management 
of  several  key  complex  roles  that  require  both  substantial  tech¬ 
nical  training  and  a  working  level  knowledge  of  the  acquisition 


Frequently,  systems  engineers  start  in  one  discipline  (such  as 
mechanical,  electrical,  or  structural),  then  learn  how  systems  in 
their  field  interact  with  other  systems  in  complex  equipment. 


systems  and  interrelationships,  systems  engineers  become 
essential  in  providing  acquisition  programs  guidance  on  how 
to  handle  changes  during  the  life  cycle  of  an  asset.  These 
may  be  subtle  changes,  such  as  cost  changes  or  equipment 
performance  characteristic  variations,  or  massive  changes 
involving  replacement  of  one  or  more  entire  systems.  Ac¬ 
curate  and  efficient  determination  of  cost,  logistics  support, 
overall  performance,  and  similar  impacts  of  changes  for  the 
program  manager  can  play  a  major  role  in  overall  success 
of  a  program. 

Despite  their  importance  to  the  acquisition  process  and  overall 
engineering  health  of  DoD,  the  health  and  relevancy  of  the 
technical  skill  level  of  personnel  in  key  roles  such  as  design 
engineer  and  systems  engineer  is  not  collectively  monitored. 
Both  roles  typically  have  no  formal  structured  technical  train¬ 
ing  within  the  government  to  capture  the  technical  level  of 
individual  practitioners  within  the  discipline.  There  are  no  for¬ 
mal  metrics  to  provide  managers  a  measure  of  the  skill  level  of 
groups  of  practitioners  within  a  branch,  division,  or  command. 
There  also  is  no  means  of  technical  knowledge  management 
for  either  role  that  could  compare  to  the  knowledge  manage¬ 
ment  method  provided  by  the  online  services  of  DAU.  Knowl¬ 
edge  in  both  systems  engineering  and  design  engineering  Is 
acquired  through  direct  experience,  individual  Investigation, 
and  direct  mentorship  from  more  experienced  personnel. 

While  these  methods  all  have  positive  attributes,  they 
also  often  lead  to  an  inconsistent  technical  message  going 


process.  There  is  sufficient  risk  in  loss  of  these  skill  sets  to 
warrant  a  structured  in-house  curriculum  to  add  order  to  a 
currently  chaotic  experiential  learning  process  associated  with 
various  on-the-job  engineering  assignments. 

Management  of  the  development  and  status  of  these  roles 
needs  to  Include  a  monitored  and  structured  developmental 
process,  have  measurable  milestones,  and  permit  the  com¬ 
mand  to  capture  the  technical  health  of  its  personnel  in  key 
roles  within  the  acquisition  community  at  any  time.  The  acqui¬ 
sition  community  needs  engineers  who  offer  a  well-balanced 
technical  perspective,  do  not  allow  the  right  process  to  drive 
them  toward  a  bad  technical  decision,  and  who  can  offer  ac¬ 
quisition  guidance  in  a  clear  and  succinct  form.  This  requires 
more  control  of  the  development  process. 

Similarly,  management  of  these  roles  must  include  capturing 
and  managing  the  associated  knowledge  in  a  manner  that 
permits  easy  access  and  a  consistent  technical  message  for 
delivery  to  developing  engineers.  One  method  of  both  con¬ 
trolling  development  and  managing  knowledge  is  to  create  a 
supportable  and  well-maintained  online  training  and  knowl¬ 
edge  management  system,  similar  to  that  used  by  DAU.  This 
will  enable  the  technical  side  of  the  partnership  between  the 
acquisition  and  technical  communities  to  function  consistently 
when  supporting  acquisition  programs  in  meeting  future  DoD 
acquisition  challenges. 

The  author  can  be  contacted  at  robert.galway^navy.mil. 
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Coaching  for  Better 


(Software)  Buying  Power  •  ;■ . 
in  an  Agile  World  * 


n  Nov.  13, 2012,  Under  Secretary  of  Defense  for  Acquisition,*Technology  and  Logistics 
Frank  Kendall  unveiled  his  guidance  for  improving  efficiency  and  productivity  under  the 
Better  Buying  Power  initiative.  Better  Buying  Power  2.0  (BBP  2.0)  identifies  36  initia¬ 
tives  under  seven  focus  areas  with  all  being  applicable  to  the  acquisition  of  software 
and  systems. 

The  extension  of  Better  Buying  Power,  coupled  with  ongoing  initiatives  to  improve  the  acquisition  of  information 
technology  (for  example,  Defense  Science  Board  2009,  Section  804  of  the  2010  Defense  Authorization  Act),  should 
lead  acquisition  professionals  carefully  to  consider  incorporation  of  agile  methodologies  into  the  set  of  acquisition 
tools  at  their  disposal.  This  transformation  is  not  easy.  It  requires  a  change  in  how  we  look  at  programs.  Therefore, 
DoD  should  consider  the  use  of  "Agile  Coaches"  to  assist  in  this  shift.  Ideally,  the  agile  coaching  corps  should  be 
internally  grown  and  assigned  to  acquisition  organizations  with  a  cadre  centrally  located  as  DAU  consultants  avail¬ 
able  to  support  any  and  all  acquisition  programs. 


Brown  has  more  than  20  years  of  defense  acquisition  experience  and  brings  both  the  contractor  and  government  perspective  to  his  cur¬ 
rent  role.  He  holds  both  Program  Management  Professional  (PMP)  and  Agile  Certified  Practitioner  (ACP)  certifications  from  the  Program 
Management  Institute. 
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This  transformation  is  not  eaTsy. 
It  requires  a  change  in  how  we^ 
look  at  programs.  Therefo^^ 

I 

I 

DoD  should  consider  the 
use  of  'Wgile  Coaches"  fo  • 
assist  in  this  shift. 


The  12  agile  principles  supporting  the  "Agile  Manifesto"  are 
closely  aligned  to  six  of  the  seven  focus  areas  of  BBP  2.0  as 
shown  in  the  following  paragraphs. 

Achieve  Affordable  Programs 

Agile  places  the  highest  priority  on  satisfying  the  customer 
through  early  and  continuous  delivery  of  valuable  software. 
Unlike  the  traditional  waterfall  method  that  delivers  at  the  end 
of  a  long  process  subject  to  schedule  delays  and  cost  over¬ 
runs,  the  agile  process  focuses  on  delivering  the  customer's 
top  priority  functionality  first  and  continuing  to  update  devel¬ 
opment  plans  so  that  the  customer  continuously  gets  what 
he  or  she  wants  the  most.  From  an  affordability  perspective, 
the  agile  process  stays  within  cost  and  schedule  constraints 
but  varies  the  features  delivered  within  those  constraints.  An 
agile  program  can  be  stopped  as  the  planned  funding  limits  or 
timeframes  are  reached,  and  the  customer  will  have  already 
received  the  most  valuable  set  of  capabilities. 

Cost  Controls  Throughout  the  Product  Life  Cycle 

In  addition  to  the  focus  on  satisfying  the  customer  described 
above,  three  additional  agile  principles  support  this  focus  area. 
Agile  methodologies  call  for  continuous  attention  to  technical 
excellence  and  good  design  to  enhance  agility.  Agile  under¬ 
stands  that  these  factors  cannot  be  completely  designed  up 
front.  Instead,  agile  processes  address  technical  excellence 
and  good  design  throughout  the  design,  development,  test, 
and  support  phases  of  the  product  life  cycle.  Agile  methods 
include  the  concept  of  technical  debt,  which,  simply  stated,  is 
the  increased  cost  of  change  due  to  poor,  inefficient  code  or 


due  to  the  backlog  of  unresolved  defects  allowed  in  the  sys¬ 
tem.  A  good  agile  team  will  plan  on  regular  refactoring  efforts 
to  ensure  that  the  software  conforms  to  high  standards.  Agile 
also  focuses  on  frequent  and  regular  delivery  of  functioning 
software  reflecting  top  user  priorities.  Therefore,  it  is  easy  to 
equate  costs  to  functionality  and  user  value  throughout  the 
development  process.  A  key  principle  of  agile  Is  simplicity,  de¬ 
fined  as  the  value  of  the  work  not  done.  The  concept  of  time¬ 
boxing  when  coupled  with  the  principle  of  simplicity  focuses 
agile  development  on  delivering  functionality  the  user  asks 
for— no  more,  no  less. 

Incentivize  Productivity  and  Innovation  in  Industry 
and  Government 

Agile  methodologies  support  the  close  alignment  between 
profitability  and  DoD  goals  through  the  frequent  delivery  of 
working  software  that  address  top  warfighter  priorities.  In  an 
agile  environment,  incentives  can  be  directly  tied  to  the  early 
and  regular  delivery  of  working  software.  Agile  also  focuses  on 
harnessing  change  for  the  customer's  advantage.  This  means 
that,  as  acquisition  professionals,  we  need  to  assess  how  and 
when  requirements  are  set  in  concrete.  This  also  means  we 
need  to  think  about  how  we  contract  for  capabilities.  In  an  agile 
world,  we  can  think  in  terms  of  fixed-price  or  fixed-price  incen¬ 
tive  contracts  if  we  fix  the  price  of  a  sprint  (using  an  agile  term 
for  a  short  duration  iteration)  then  buy  a  number  of  sprints 
as  an  option  if  the  contractor  meets  promised  velocity.  This 
strategy  allows  both  the  government  and  the  contractor  to 
be  responsive  to  changes  in  requirements  or  user  priorities. 

Eliminate  Unproductive  Processes  and 
Bureaucracy 

Agile  has  a  principle  called  simplicity  that  is  the  art  of  maximiz¬ 
ing  the  amount  of  work  not  done,  that  captures  the  essence 
of  this  focus  area.  Traditional  information  technology  acqui¬ 
sition  follows  a  sequential,  stovepiped  waterfall  process  that 
results  in  a  large  amount  of  "work  in  progress"  throughout 
the  development  effort  and  delays  delivery  of  functionality 
to  the  end  user  until  the  end  of  the  effort.  Agile  believes  that 
DevOps,  the  process  of  warfighters  and  developers  work¬ 
ing  together  throughout  the  project,  is  superior  to  volumes 
of  detailed  documentation  subject  to  misinterpretation— or 
worse,  nonuse— and  results  in  early  and  frequent  delivery  of 
the  capabilities  the  user  needs.  Simplicity  also  comes  into  play 
here  in  that  developers,  working  closely  with  warfighters,  can 
accurately  identify  when  a  capability  is  good  enough.  If  20 
percent  of  the  effort  can  deliver  80  percent  of  the  functionality 
and  the  warfighter  is  happy,  the  Department  is  better  served 
if  the  remaining  funds  are  allocated  where  they  can  address 
the  most  pressing  needs. 

Promote  Effective  Competition 

This  starts  with  establishing  the  government  as  the  product 
owner  and  ensuring  that  the  government  owns  the  functional 
and  technical  vision.  For  afloat  forces  in  the  Navy,  we  ex¬ 
pect  software  capabilities  to  ride  on  the  Consolidated  Afloat 
Networks  and  Enterprise  Services  (CANES)  infrastructure. 
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Contractors  follow  well-defined  software  development  stan¬ 
dards  designed  to  promote  interoperability  and  avoid  propri¬ 
etary  code  from  creeping  into  deployed  systems.  It  also  means 
the  government  program  offices  need  to  clearly  define  the 
desired  government  technical  data  rights. 

Improve  the  Professionalism  of  the  Total 
Acquisition  Workforce 

The  DoD  needs  to  invest  in  training  the  acquisition  workforce 
in  agile  methodologies  to  add  tools  that  can  be  used  effec¬ 
tively  when  appropriate.  The  Department  also  needs  to  look 
at  organizational  structures  and  relationships  that  promote  the 
values  identified  in  BBP  2.0.  Operational  commands  need  to 
understand  their  role  In  defining  priorities  and  working  with 
the  acquisition  agencies  to  ensure  their  voice  is  heard  through¬ 
out  the  development  process.  One  of  the  most  fundamental 
changes  is  in  how  the  Department  manages  requirements— or, 
as  they  are  referred  to  in  an  agile  environment,  features.  The 
incremental,  iterative  evolution  of  requirements  throughout 
the  life  of  a  project  calls  for  active  participation  instead  of  the 
frequent  practice  of  throwing  the  requirements  over  the  fence 
and  waiting  years  for  results.  Acquisition  professionals  need 
to  learn  how  to  manage  features  (large  blocks  of  functional¬ 
ity)  and  user  stories  (detailed  requirements)  instead  of  tasks. 
At  senior  levels,  we  need  to  think  about  how  we  manage  and 
assess  the  effectiveness  of  investment  strategies. 

In  the  preceding  paragraphs  I  discussed  how  agile  principles 
support  the  BBP  2.0  focus  areas.  Now  I  want  to  focus  on  the 
transformation  to  agile. 

The  first  question  always  is,  "Why?"  Version  One,  a  leading 
provider  of  tools  supporting  agile  development,  conducts 
an  annual  State  of  Agile  Survey.  The  results  for  2011,  based 
on  more  than  6,000  responses,  indicated  that  the  ability  to 
manage  changing  customer  priorities  (cited  on  84  percent  of 
respondents)  replaced  improved  productivity  (75  percent) 
as  the  leading  reason  to  be  agile.  Equally  surprising  was  the 
fact  that  project  visibility  (77  percent)  moved  into  the  second 
position,  indicating  that  senior  decision  makers  are  getting  the 
information  they  need  from  agile  projects. 

A  number  of  agile  programs  currently  are  being  executed 
within  the  DoD  environment.  The  DoD  even  has  a  draft 
Agile  Handbook  (Mitre  Technical  Report  100489)  that 
identifies  both  the  advantages  and  barriers  a  program 
faces  as  it  tries  to  adopt  agile  methodologies.  The  Govern¬ 
ment  Accountability  Office  (see  GAO  Report  12-681)  was 
asked  to  "Identify  (1)  effective  practices  in  applying  agile 
for  software  development  solutions  and,  (2)  federal  chal¬ 
lenges  in  implementing  Agile  development  techniques." 
Both  documents  contain  specific  recommendations  that 
address  key  reasons  agile  projects  fail  if  the  Department 
considers  moving  toward  increased  use  of  agile  software 
development  methodology.  The  2011  State  of  Agile  Survey 
reported  that  the  leading  causes  of  failure  for  agile  projects 
were  lack  of  experience  In  agile  methodologies  and  failure  to 


In  an  agile  world,  we  can 
think  in  terms  of  fixed-price 
or  fixed-price  incentive 
contracts  if  we  fix  the  price  of 
a  sprint  (using  an  agile  term 
for  a  short  duration  iteration) 
then  buy  a  number  of  sprints 
as  an  option  if  the  contractor 
meets  promised  velocity. 


understand/address  the  broader  organizational  Issues  in¬ 
volved.  These  top  two  were  followed  closely  by  corporate 
culture  issues  and  pressures  for  traditional  waterfall  methods. 

Agile  reflects  a  way  of  thinking  about  executing  projects 
that  cannot  be  learned  in  one  or  two  classes.  The  Defense 
Acquisition  University  (DAU)  is  continuously  improving  the 
Information  technology  curriculum.  The  challenge  will  be  to 
continually  raise  the  bar  as  agile  approaches  reach  deeper 
and  broader  into  the  national  defense  system.  DAU  also 
offers  coaching  services  but  the  number  of  agile  certified 
coaches  is  currently  limited.  These  are  valuable  resources. 
However,  software  development  organizations  should  con¬ 
sider  establishing  an  Internal  Agile  Coach  position  within 
the  acquisition  and/or  program  management  competency 
to  provide  the  ongoing  mentoring  and  advice  to  individual 
programs  considering  adopting  agile.  Both  of  the  previously 
referenced  documents  recommend  training  and  the  involve¬ 
ment  of  an  Agile  Coach  to  help  projects  and  organizations 
align  processes  and  organizational  structures  to  support 
agile  methods.  The  Agile  Coach  also  should  help  the  or¬ 
ganization  address  the  transition  from  traditional  waterfall 
processes  toward  increased  agility.  In  BBP  2.0,  Mr.  Kendall 
challenged  every  acquisition  professional  to  do  "more  with 
less."  Agile  methodologies  may  provide  the  means  for  us 
to  meet  that  challenge. 


The  author  can  be  contacted  at  martin. brownl^navy.mil. 
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he  TEMP,  or  Test  and  Evaluation 
Master  Plan,  is  the  framework 
for  an  acquisition  program's 
Test  and  Evaluation  (T&E)  pro¬ 
gram.  The  TEMP  is  a  four-part 
critical  program  document  that 
links  directly  to  the  Acquisition 
Strategy  and  the  System  Perfor¬ 
mance  Specification.  The  TEMP 
shows  how  the  program  will 
verify  and  validate  the  system 
requirements,  whereas  the  Ac¬ 
quisition  Strategy  speaks  to  the 
management  of  the  acquisition 


Conroy  is  a  professor  of  systems  engineering  (test  and  evaluation)  in  the  Capital  and  North¬ 
east  Region  of  the  Defense  Acguisition  University  at  Fort  Belvoir,  Va. 
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of  the  requirements,  and  the  System  Performance  Specifica¬ 
tion  guides  the  development  of  those  requirements.  The  TEMP 
has  a  crucial  role  in  ensuring  that  the  system  meets  the  users' 
requirements  and  capabilities. 

Each  of  the  TEMP's  four  parts  is  integral  to  answering  the 
"why"  questions  surrounding  the  programming  and  planning 
for  the  developmental  test  (DT)  and  operational  test  (OT)  and 
evaluation  methods  and  resources.  If  the  TEMP  is  written  cor¬ 
rectly,  the  order  of  the  four  parts  also  tells  a  story  and  answers 
these  "why"  questions  effectively.  If  these  "why"  questions  are 
used  when  creating  a  TEMP,  it  will  be  a  very  useful  document 
for  managing  the  test  program. 

Parti 

The  TEMP  has  four  main  parts.  Part  I  of  the  TEMP  is  called 
the  Introduction,  but  in  reality  it  is  everything  one  needs  to 
know  about  the  system  being  developed,  tested,  and  evalu¬ 
ated.  The  relevant  question  answered  by  the  information  in 
Part  I  is  "Why  is  this  system  needed?"  One  can  see  that  Part  I 


delve  into  this  part,  let's  talk  a  little  bit  about  the  two  views  of 
testing  in  terms  of  evaluated  performance  and  the  two  views 
of  testing  in  terms  of  evaluation  focus. 

When  testing  the  performance  of  a  system,  the  system  is 
tested  and  evaluated  for  effectiveness  and  suitability.  Effec¬ 
tiveness  is  the  ability  of  the  system  to  meet  its  mission  and 
suitability  is  the  ability  of  the  system  to  be  available  to  meet 
its  mission.  That  is  it  in  a  nutshell.  There  are  more  detailed 
definitions,  but  those  are  the  basics.  An  example  would  be 
that  the  effectiveness  of  a  car  is  that  it  has  the  ability  to  get 
you  to  your  destination  within  your  timeframe,  whereas  the 
suitability  of  a  car  is  that  it  is  reliable  and  ready  to  drive  and 
that  it  can  be  driven.  If  the  car  can  get  you  to  your  destinations 
but  you  need  to  change  the  oil  each  trip,  it  may  be  effective 
but  not  very  suitable. 

Having  said  that,  let's  discuss  the  focus  of  test  and  evalua¬ 
tion.  The  two  main  views  of  evaluation  are  from  the  points  of 
developmental  testing  and  operational  testing.  These  views 


At  the  end  of  the  developmental  test 
program,  we  do  not  want  to  know  that  the  system  works 
well  in  a  lab  or  controlled  environment;  we  want  to  know  what  to 
expect  when  operating  the  system  in  the  real  environment. 


answers  this  question  with  the  background  information  about 
the  system  and  what  capabilities  and  requirements  are  neces¬ 
sary  to  achieve  Its  mission.  Part  I  also  uses  this  information  to 
explain  the  rationale  behind  the  prioritization  of  the  capabilities 
and  requirements  for  the  system  by  explaining  the  nature  of 
the  threat  and  how  the  system  combats  it. 

Part  II 

Part  II  of  the  TEMP  is  known  as  the  Test  Program  Manage¬ 
ment  and  Schedule.  This  section  is  very  straightforward  and  it 
answers  the  primary  "why"  question  of  "Why  does  this  testing 
need  to  be  done  now  and  under  this  budget?"  This  is  important 
because  it  will  constrain  the  amount  of  testing  and  evaluation 
that  can  be  done  on  the  program  to  prove  that  the  system 
is  effective  and  suitable  in  meeting  Its  objectives.  We'll  talk 
more  about  what  it  means  to  be  effective  and  suitable  in  Part 
III.  Part  II  sets  the  boundaries  within  which  the  test  program 
needs  to  be  accomplished  successfully.  This  will  be  significant 
when  trying  to  establish  the  best  tradeoffs  between  how  much 
testing  is  desired  and  how  much  testing  is  needed  to  evaluate 
what  can  be  expected  of  the  system's  true  performance  when 
used  in  the  field. 

Part  III 

The  next  section  is  Part  III,  the  Test  and  Evaluation  Strategy. 
This  section  is  the  heart  and  soul  of  the  TEMP.  But  before  we 


used  to  be  very  diverse,  so  much  so  that  what  is  now  Part 
III  once  was  two  separate  sections,  one  for  developmental 
testing  and  one  for  operational  testing.  Both  views  are  now 
integrated  into  Part  III. 

Developmental  testing  focuses  on  giving  you  what  you  asked 
for.  It  answers  the  question  "Did  I  build  it  right?"  Developmen¬ 
tal  testing  is  more  to  the  point  of  meeting  the  requirement,  or 
what  was  asked  for,  while  trying  to  meet  the  needed  capability. 
However,  if  the  needed  capability  was  not  correctly  translated 
into  a  specified  requirement,  then  what  was  asked  for  may  not 
meet  that  need.  This  second  view,  which  answers  the  ques¬ 
tion  "Did  I  build  the  right  thing?"  is  called  validation  and  is 
the  focus  of  operational  testing.  It  Is  easy  to  see  how  the  two 
can  diverge  if  the  translated  need  is  not  fully  resolved  by  the 
stated  requirements.  One  example  may  be  to  state  the  need 
for  a  200-square-foot  room.  If  this  is  the  only  requirement, 
the  requirement  can  be  met,  or  pass  verification  and  thereby 
developmental  testing,  by  any  combination  of  square  footage 
in  the  room  that  totals  200.  However  a  room  that  is  2  feet  wide 
by  100  feet  long  may  not  suit  your  needs  and  would  not  meet 
validation  or  operational  testing. 

One  can  see  how  important  it  is  that  developmental  testing 
and  operational  testing,  or  verification  and  validation,  are 
given  their  due  in  supporting  each  other  to  gain  the  end  user 
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a  system  that  meets  all  the  requirements  and  capabilities  to 
be  both  effective  and  suitable  in  the  field.  To  this  end,  the  op¬ 
erational  test  community  focuses  heavily  on  integrated  test 
and  evaluation.  Integrated  test  and  evaluation  involve  the  in¬ 
tegration  of  developmental  testing  with  operational  testing. 
This  is  accomplished  in  many  ways,  but  one  of  the  best  ways 
is  to  make  developmental  tests  look  and  feel  like  operational 
tests  as  much  as  possible.  At  the  end  of  the  developmental 
test  program,  we  do  not  want  to  know  that  the  system  works 
well  in  a  lab  or  controlled  environment;  we  want  to  know  what 
to  expect  when  operating  the  system  in  the  real  environment. 
To  do  this,  developmental  testing  environments  need  to  be 
instituted  to  the  greatest  extent  possible  to  simulate  increasing 
levels  of  the  operational  environment,  thereby  decreasing  the 
risk  over  the  test  program  on  the  way  to  a  fielding  decision. 

This  is  ultimately  why  there  is  a  single  combined  developmen¬ 
tal  and  operational  test  focus  in  Part  111  to  reach  both  effective¬ 
ness  and  suitability.  Verification  must  work  with  validation,  and 
effectiveness  must  be  balanced  with  suitability.  Part  ill  brings 
all  these  together  to  explain  the  test  and  evaluation  strategy  as 
a  whole  to  include  how  many  tests  it  will  take,  what  methods 
of  test  and  evaluation  are  necessary  for  each  requirement  and 
capability,  and  how  the  complete  program  balances  to  meet 
the  need.  Ultimately,  Part  111  answers  the  "why"  question  of 
"Why  is  this  combination  of  tests  necessary  to  evaluate  the 
system's  performance?" 

Part  IV 

Part  IV  is  the  final  part  of  the  TEMP  and  it  is  called  the  Re¬ 
source  Summary.  This  is  the  point  everything  else  was  leading 
up  to.  This  is  what  gets  the  plan  done.  Part  IV  is  the  description 
of  the  resources  in  terms  of  funding,  test  sites,  and  test  assets 


that  will  be  needed  to  meet  the  test  and  evaluation  strategy 
described  in  Part  IN. 

Part  IV  is  the  end  of  the  document  but  also  a  beginning  in 
terms  of  evaluating  the  TEMP  to  see  if  it  is  effective  as  a  plan¬ 
ning  tool  for  the  program  once  it  has  been  written.  If  you  ask 
"why"  of  each  part,  you  should  be  able  to  find  the  answer  in 
the  previous  part  and  be  able  to  work  your  way  back  through 
the  TEMP  with  all  your  questions  answered.  If  you  ask  "Why 
are  these  resources  in  Part  IV  needed  to  accomplish  this  test 
program?,"  you  should  be  able  to  find  all  the  answers  in  terms 
of  what  tests  depend  on  those  resources  in  Part  IN.  If  you  ask 
"Why  are  these  tests  constrained  the  way  they  are  in  Part  1 1 1?," 
you  should  be  able  to  find  those  answers  in  Part  11.  And  if  you 
ask  "Why  do  these  tests  in  Part  1 1 1  need  to  be  conducted?,"  you 
should  be  able  to  find  those  answers  in  Part  1.  Finally,  if  you  ask 
"Why  is  the  program  constrained  the  way  it  is  in  Part  11?,"  you 
should  be  able  to  find  those  answers  in  Part  1. 

Summary 

The  four-part  TEMP  is  an  effective  tool  in  planning  the  test 
and  evaluation  program  for  a  system  in  development.  The 
TEMP  has  a  crucial  role  in  ensuring  that  the  system  meets  the 
users'  requirements  and  capabilities  that  are  documented  in 
the  System  Performance  Specification  and  acquired  and  man¬ 
aged  through  the  Acquisition  Strategy.  It  is  a  document  that 
answers  a  number  of  questions  about  the  nature  of  the  test 
and  evaluation  program.  In  answering  those  questions  while 
developing  the  TEMP,  the  TEMP  becomes  more  effective  as  a 
management  and  planning  tool  supporting  the  entire  system 
acquisition  and  management  program.  When  it  comes  to  the 
TEMP,  it  is  OK  to  keep  asking  "why." 

The  author  can  be  contacted  at  Tom.Conroy@dau.mn. 
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Values  Take  Center  Stage 


Stan  Emelander 


The  greatest  good  of  a  man 
is  daily  to  converse  about  virtue, 
and  all  that  concerning  which  you 
hear  me  examining  myself  and 
others,  and  that  the  life  which  is 
unexamined  is  not  worth  living. 


For  the  past 
33  years  I  have 
looked  in  the  mirror 
and  asked  myself,  "If  today 
were  the  last  day  of  my  life, 
would  I  want  to  do  what  I  am 
about  to  do  today?" 


—Socrates,  399  BC 


—Steve  Jobs,  2005  AD 


Values  exert  a  powerful  influence  on  our  behavior,  whether  or  not  we  deliberately  choose 
which  are  most  important.  I  think  this  is  primarily  what  Socrates  meant.  Consider  how 
you  start  your  day.  Let's  say  you  are  someone,  like  me,  who  sometimes  has  trouble 
getting  going  in  the  morning.  As  you  hustle  through  your  morning  routine,  you  might 
feel  pressed  for  time,  a  little  pressured  and  hassled.  What  are  your  concerns  when 


Emelander  is  o  product  manager  in  the  Army's  Individual  Weapons  program.  He  holds  a  doctoral  degree  in  organization  and  management 
and  is  a  graduate  of  the  Excellence  in  Government  Fellowship  sponsored  by  the  Partnership  for  Public  Service.  He  is  Level  II  certified  in  Program 
Management  and  Level  I  in  Systems  Engineering. 
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The  ideal  situation  exists  when  there  is  congruence 
between  individual  and  organizational  values,  often 
embodied  by  the  firm's  leaders. 


you  feel  this  way?  Are  you  in  control?  Where  are  your  values? 
Do  you  regularly  start  your  day  in  a  positive  or  negative  frame 
of  mind?  Think  about  the  many  other  circumstances  where 
your  mood,  outlook,  and  effectiveness  are  influenced  by  values 
that  may  be  unconscious  and  out  of  control. 

In  my  experience,  values  are  universally  recognized  as  impor¬ 
tant,  but  also  often  weakly  understood  and  acted  upon.  As  a 
starting  point,  it  helps  to  define  the  term  clearly.  Values  can 
be  defined  as  deeply  held  beliefs  and  needs  that  guide  our 
decisions  and  behavior,  the  principles  and  standards  that  give 
meaning  to  life.  Core  values  are  those  we  will  not  violate,  even 
when  the  stakes  are  high.  The  concept  of  values  also  extends 
to  our  personal  likes,  dislikes,  and  preferences.  For  instance, 
although  a  desire  to  exercise  every  day  may  seem  unrelated 
to  deeply  held  moral  beliefs,  it  cannot  be  rejected  as  a  value. 
If  being  fit  contributes  to  your  quality  of  life,  and  you  deeply 
enjoy  the  activity,  exercise  also  has  a  place  among  your  values. 
Values,  then,  both  constrain  our  behavior  and  compel  us  to 
take  action. 

Why  Values  Matter 

Developing  and  acting  on  strong  values  is  important  for  pro¬ 
fessional  success  and  personal  meaning.  It's  unsurprising 
that  studies  show  people  who  recognize  and  regularly  act 
on  their  core  values  experience  greater  fulfillment,  satisfac¬ 
tion,  and  success.  Personal  values  can  become  meaningful 
goals,  and  working  to  achieve  substantial  goals  is  the  prime 
ingredient  of  a  purposeful  life.  Studies  show  that  those  who 
prize  intrinsic  values,  such  as  meaningful  work,  experience 
greater  happiness  than  those  who  esteem  extrinsic  values 
like  wealth,  even  when  the  differences  in  wealth  are  large. 
The  integration  of  values  with  work  is  one  way  to  answer 
this  question:  "Do  you  work  to  do  something,  or  for  some¬ 
thing  to  do?" 

The  case  for  strong  organizational  values  is  just  as  convinc¬ 
ing.  Organizational  values  can  point  the  way  to  behaviors  that 
power  the  firm's  strategy,  such  as  creative  risk-taking  or  put¬ 
ting  customer's  needs  first.  Trust,  to  take  one  value,  has  been 
identified  as  the  key  distinguishing  feature  of  top-performing 
business.  Research  also  supports  the  link  between  commit¬ 
ted  workers  and  business  success,  making  the  firm's  support 
for  workers'  value  fulfillment  a  top  priority.  The  ideal  situa¬ 
tion  exists  when  there  is  congruence  between  individual  and 
organizational  values,  often  embodied  by  the  firm's  leaders. 
Values  also  play  an  essential  role  in  leadership  development. 


In  the  field  of  leadership  studies,  values  are  strongly  associ¬ 
ated  with  the  greatest  role  models.  True  leadership  can  be 
thought  of  as  the  art  of  persuading  others  to  act  when  they  can 
choose  not  to,  and  the  strongest  call  to  action  often  originates 
from  a  leader's  values.  Aspiring  leaders  everywhere  identify 
role  models,  including  Abraham  Lincoln,  Nelson  Mandela, 
Mahatma  Gandhi,  George  Washington,  and  others  whose 
strongly  held  and  effectively  communicated  values  contrib¬ 
uted  to  their  profound  impact.  While  there  are  many  differ¬ 
ent  leadership  styles  and  behaviors,  nothing  is  stronger  than 
matching  values  in  persuading  followers  to  act.  However,  de¬ 
spite  their  appeal  and  importance,  multiple  challenges  inhibit 
developing  and  acting  upon  values. 

Barriers  to  Value  Formation 

If  strong  personal  values  are  such  a  potent  force  for  success 
and  fulfillment,  what  blocks  their  development  and  implemen¬ 
tation?  We  can  identify  several  factors.  Deciding  upon  a  core 
set  of  values  might  be  a  daunting  task,  requiring  consider¬ 
able  Introspection,  and  finding  time  for  self-reflection  can  be 
a  challenge  in  our  era  of  expanding  job  hours  and  constant 
information  input.  Analysis  of  one's  own  behaviors  can  be 
uncomfortable,  especially  if  when  we  are  asking  unfamiliar 
questions.  Also,  the  sheer  number  of  values,  and  their  inter¬ 
relatedness,  complicates  the  task.  Trust,  for  example,  can  be 
thought  of  as  consisting  of  reliability  and  competence,  which 
are  themselves  values.  So  what  does  it  really  mean  to  hold 
trustworthiness  as  a  value? 

Additional  inhibitors  exist  in  teams  and  other  organizations.  At 
work,  values  may  be  considered  a  personal  matter,  something 
we  are  reluctant  to  discuss.  When  is  the  last  time  you  asked 
your  supervisors  about  their  values?  When  have  you  explained 
your  values  to  your  constituents?  Barriers  to  communication, 
including  clarity,  frequency,  and  information  overload  can 
hinder  the  distribution  of  leaders'  Intent  concerning  values. 
An  effort  to  instill  a  new  set  of  values  can  entail  a  change  to 
organizational  culture  that  may  be  resisted.  Factors  such  as 
fear,  a  perceived  threat  to  power  and  prestige,  and  fatigue 
from  past  change  efforts  must  be  overcome  for  new  cultural 
values  to  take  hold.  There  also  is  the  potential  problem  of  over¬ 
exposure  leading  to  cynicism  and  a  "flavor  of  the  moment" 
attitude  on  the  part  of  workers.  Another  challenge  is  conflict 
between  organizational  and  individual  values,  leading  to  con¬ 
fusion.  Employees  are  quick  to  detect  discrepancies  between 
the  organization's  stated  values  and  conflicting  behaviors  by 
leaders  at  any  level. 
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Discovering  Values 

You  may  have  to  dive  deep  to  retrieve  your  values.  I  was  initially 
overwhelmed  trying  to  sift  through  lists  of  possible  values,  but 
started  making  progress  when  I  followed  some  of  the  guide¬ 
lines  available  on  the  Internet  and  elsewhere.  These  Included 
imagining  one's  own  memorial  service  and  what  one  would 
most  want  to  be  remembered  for,  and  thinking  about  what 
made  peak  experiences  so  significant:  First  identify  some  peak 
experiences,  then  recall  your  feelings  at  that  time  and  since. 
Another  method  is  self-observation,  reflecting  on  your  regular 
behavior.  Your  repeated  activities,  encompassing  all  areas  of 
life,  signal  where  your  values  lie.  Whatever  the  method  used, 
record  the  values  you  identify  and  make  a  date  with  yourself 
to  revisit  them.  I  find  that  my  most  important  values  change, 
becoming  clearer,  when  I  consider  them  again. 

Collective  values  may  be  developed  for  teams,  departments, 
and  the  whole  enterprise.  Along  with  mission  and  vision  state- 


well  do  they  match  your  ideal  list  of  values?  This  exercise  can 
be  done  for  the  various  spheres  of  life,  including  your  work  life. 

Another  method  focuses  on  enacting  your  values,  thinking  of 
ways  you  can  put  your  values  into  action.  Research  the  defi¬ 
nition  of  each  of  your  values  and  write  an  expanded  personal 
definition,  concentrating  on  how  it  could  be  enacted  and  what 
specific  behaviors  make  it  come  to  life.  Select  a  value  to  enact 
and  focus  upon  each  day;  if  your  day  includes  meetings  or 
other  trying  activities,  attempt  to  be  specifically  conscious  of 
your  value  intent  throughout  the  event.  Yet  another  technique 
is  to  identify  an  icon  that  represents  your  values.  It  might  be 
a  person,  an  animal  (lion,  eagle),  or  something  drawn  from 
nature  (ocean,  mountain).  Periodically  touch  bases  with  your 
Icon,  especially  when  you  feel  pressured. 

In  addition  to  their  personal  values,  managers  and  leaders 
must  consider  how  to  empower  organizational  values.  The 


There  also  is  the  potential  problem  of  overexposure 
leading  to  cynicism  and  a  “flavor  of  the  moment"  attitude 

on  the  part  of  workers. 


ments,  a  statement  of  core  values  Is  a  common  method  man¬ 
agers  use  to  communicate  organizational  purpose  and  influ¬ 
ence  culture.  Group  values  have  multiple  sources,  including  the 
parent  enterprise,  the  top  management  team,  and  customers, 
but  the  most  potent  source  of  an  organization's  values  is  usu¬ 
ally  Its  employees.  Effective  leaders  are  in  touch  with  followers 
and  pay  attention  to  their  dreams  and  aspirations,  fostering 
organizational  values  that  share  an  organic  relationship  with 
employee  values.  These  have  the  strongest  resonance  and  are 
more  likely  to  be  adopted  as  part  of  the  organization's  culture. 
Transformational  leadership,  including  effective  communica¬ 
tion  of  an  empowering  vision,  especially  seeks  to  address  gaps 
or  dissonance  between  individual  and  collective  values. 

Empowering  Values 

A  first  step  to  empowering  your  values  is  to  analyze  how  well 
you  are  now  enacting  them.  Ideally,  your  behavior  should 
match  your  values,  with  the  most  time  and  intensity  devoted 
to  those  most  important.  One  straightforward  means  to  weigh 
this  alignment  is  to  draw  a  line  down  the  middle  of  a  piece  of 
paper.  On  the  left  side  list  your  regular  activities,  on  the  right 
side  your  reasons  for  these  behaviors.  If  one  activity-reason 
pair  (like  servicing  your  car)  is  in  support  of  another  (like  get¬ 
ting  to  work)  cross  it  out,  so  only  root  activities  and  reasons 
remain.  Look  at  the  reasons  for  the  remaining  behaviors.  How 


organization's  values  (conscious  or  unconscious)  are  at  the 
heart  of  what  it  does  to  survive  (i.e.,  its  strategy),  and  enacting 
those  values  also  Is  at  the  heart  of  a  leader's  role.  Two  aspects 
of  leadership  enable  this  effort:  effective  communications  and 
role  modeling.  Discussion  of  values  is  important  and  does  not 
have  to  be  an  extraordinary  event.  It  is  reasonable,  for  instance, 
to  emphasize  trust,  fairness,  or  honesty,  as  themes  at  the  start 
of  a  meeting,  or  to  explore  what  values  set  your  team,  depart¬ 
ment,  or  organization  apart  from  others.  This  method,  asking 
"what  makes  us  special,"  offers  leadership  opportunities  for 
employees  at  all  levels  and  lowers  the  barrier  to  value  infusion. 

Role  modeling  Is  arguably  the  most  powerful  method  at  the 
leader's  disposal  to  affect  follower  behavior  and  beliefs. 
Values  such  as  customer  service  come  alive  when  workers 
observe  leaders  helping  customers  themselves.  For  orga¬ 
nizations,  as  for  Individuals,  the  greatest  challenge  lies  in 
enacting  values.  To  avoid  the  "hollow  values"  syndrome, 
managers  must  follow  through  and  to  see  that  behavior 
and  rewards  match  the  organization's  values.  Remember, 
"What  get  measured  gets  done."  When  values  are  consis¬ 
tent  between  leaders,  followers,  and  customers,  everyone 
benefits  from  their  fulfillment. 


The  author  can  be  contacted  at  stanley.j.emelander.civ^mail.mn. 


47 


Defense  AT&L:  May-June  2013 


Our  Troops  Need  Your 

BRAINPOWER 

Here’s  a  way  to  put  it  to  work 


Join  the  best  nninds  in  science  and  technology  on  DoDTechipedia— the  new  internal  wiki  for  the 
U.S.  Departnnent  of  Defense.  Post  ideas,  ask  questions,  make  suggestions,  or  share  information 
with  colleagues  you  can’t  reach  now.  It’s  a  way  to  expand  our  brainpower,  focusing  on  rapidly 
responding  to  the  needs  of  the  warfighter. 

HERPS  HOW  IT  WORKS 

•  Share  your  knowledge.  Every  contribution  counts.  The  more 
you  contribute,  the  more  the  collective  knowledge  base  ex¬ 
pands.  The  wiki  can  easily  be  edited  by  any  user,  broadening 
your  access  to  the  latest  and  best  research  and  ideas.  DoD¬ 
Techipedia  is  open  to  federal  government  employees  and 
contractors  with  Common  Access  Card  or  DTIC  registration. 

•  Connect  across  walls.  Reach  across  command  chains  and 
departmental  divisions  to  find  other  people  working  on 


THE  INFORMATION  ASSURANCE 
TECHNOLOGY  ANALYSIS  CENTER 
(lATAC)  MAINTAINS  THE  FOLLOW¬ 
ING  TECHNOLOGY  FOCUS  AREAS: 

Information  Assurance: 

Protection  and  defense  of  information  and  IT 
systems 

Information  Warfare: 

Capabilities  used  to  exploit  information  and 
IT  systems 

Networking  Technology: 

Technologies  that  interconnect  groups  and/ 
or  systems 

lATAC  POC:  Rogelio  Raymond 
703-984-0072  or 
raymond_rogelio@bah.com 


The  U.S.  Department  of  Defense  Science  and  Technology  Wiki 

A  project  of  Acquisition,  Technology  and  Logistics,  Defense  Research  and  Engineering,  Defense  Technical  In¬ 
formation  Center,  Networks  and  Information  Integration  and  Department  of  Defense  Chief  Information  Officer, 
and  Rapid  Reaction  Technology  Office 


ideas  and  solutions  that  interest  you.  Discuss  hot  topics. 

Stay  on  top  of  new  trends.  Read  technical  blogs— or  create 
one  of  your  own.  You  don’t  need  to  know  the  right  people— 
you  can  connect  on  the  wiki. 

•  Collaborate.  The  wars  we  are  fighting  today  require  immedi¬ 
ate  solutions.  The  wiki  is  the  biggest  brainstorming  session 
ever  at  DoD.  Network  with  others  working  in  your  areas  of 
interest.  Present  new  ideas  or  technical  challenges.  Stay 
abreast  of  research  and  development  initiatives,  confer¬ 
ences,  and  symposia.  Collaboration  across  DoD  increases 
our  ability  to  identify  challenges  as  they  emerge  and  deliver 
vigorous  solutions  fast. 


https://www.DoDTechipedia.mil 


START  CONTRIBUTING 
TO  DODTECHIPEDIA  NOW 

If  you  have  CAC  or  DTIC  registration, 
you  already  have  access  to  the  wiki. 
Go  to  httpsi/ywww.DoDTechipedia.mil 
and  log  in.  Once  on  the  wiki,  visit  the 
tutorials  link  to  learn  how  to  add  or 
edit  information. 


Defense  AT&L 

Writers’  Guidelines  in  Brief 


Purpose 

Defense  AT&L  is  a  bimonthly  magazine  published  by  DAU  Press, 
Defense  Acquisition  University,  for  senior  military  personnel, 
civilians,  defense  contractors,  and  defense  industry  profession¬ 
als  in  program  management  and  the  acquisition,  technology,  and 
logistics  workforce. 

Submission  Procedures 

Submit  articles  by  e-mail  to  datl(a)dau.mil.  Submissions  must  include 
each  author's  name,  mailing  address,  office  phone  number,  e-mail 
address,  and  brief  biographical  statement.  Each  must  also  be  ac¬ 
companied  by  a  copyright  release. 

Receipt  of  your  submission  will  be  acknowledged  in  5  working  days. 
You  will  be  notified  of  our  publication  decision  in  2  to  3  weeks.  All 
decisions  are  final. 


Deadlines 

Note:  If  the  magazine  fills  up  before  the  author  deadline,  submissions 
are  considered  for  the  following  issue. 


Issue 

January-February 

March-April 

May-June 

July-August 

September-October 

November- December 


Author  Deadline 

1  October 
1  December 
1  February 
1  April 
1  June 
1  August 


Audience 

Defense  AT&L  readers  are  mainly  acquisition  professionals  serving 
in  career  positions  covered  by  the  Defense  Acquisition  Workforce 
Improvement  Act  (DAWIA)  or  industry  equivalent. 

Style 

Defense  AT&L  prints  feature  stories  focusing  on  real  people  and 
events.  The  magazine  seeks  articles  that  reflect  author  experiences 
in  and  thoughts  about  acquisition  rather  than  pages  of  researched 
information.  Articles  should  discuss  the  individual's  experience  with 
problems  and  solutions  in  acquisition,  contracting,  logistics,  or  pro¬ 
gram  management,  or  with  emerging  trends. 


The  magazine  does  not  print  academic  papers;  fact  sheets;  technical 
papers;  white  papers;  or  articles  with  footnotes,  endnotes,  or  refer¬ 
ences.  Manuscripts  meeting  any  of  those  criteria  are  more  suitable 
for  DAU's  journal.  Defense  Acquisition  Research  Journal  (ARJ). 


Defense  AT&L  does  not  reprint  from  other  publications.  Please  do  not 
submit  manuscripts  that  have  appeared  elsewhere.  Defense  AT&L 
does  not  publish  endorsements  of  products  for  sale. 


Length 

Articles  should  be  1,500-2,500  words. 

Format 

Send  submissions  via  e-mail  as  Microsoft  Word  attachments. 

Graphics 

Do  not  embed  photographs  or  charts  in  the  manuscript.  Digital  files 
of  photos  or  graphics  should  be  sent  as  e-mail  attachments.  Each 
figure  or  chart  must  be  saved  as  a  separate  file  in  the  original  soft¬ 
ware  format  in  which  it  was  created. 

TIF  or  JPEG  files  must  have  a  resolution  of  300  pixels  per  inch; 
enhanced  resolutions  are  not  acceptable;  and  images  downloaded 
from  the  Web  are  not  of  adequate  quality  for  reproduction.  De¬ 
tailed  tables  and  charts  are  not  accepted  for  publication  because 
they  will  be  illegible  when  reduced  to  fit  at  most  one-third  of  a 
magazine  page. 

Non-DoD  photos  and  graphics  are  printed  only  with  written  per¬ 
mission  from  the  source.  It  is  the  author's  responsibility  to  obtain 
and  submit  permission  with  the  article.  Do  not  include  any  clas¬ 
sified  information. 

Author  Information 

Contact  and  biographical  information  will  be  included  with  each 
article  selected  for  publication.  Please  include  the  following  infor¬ 
mation  with  your  submission:  name,  position  title,  department,  in¬ 
stitution,  address,  phone  number,  and  e-mail  address.  Also,  please 
supply  a  short  biographical  statement,  not  to  exceed  25  words.  We 
do  not  print  author  bio  photographs. 

Copyright 

All  articles  require  a  signed  Work  of  the  U.S.  Government/Copyright 
Release  form,  available  at  http://www.dau.mil/pubscats/pages/ 
defenseatl.aspx.  Fill  out,  sign,  scan,  and  e-mail  it  to  datl(a)dau.mil 
or  fax  it  to  703-805-2917,  Attn:  Defense  AT&L. 

Alternatively,  you  may  submit  a  written  release  from  the  major 
command  (normally  the  public  affairs  office)  indicating  the  author 
is  releasing  the  article  to  Defense  AT&L  for  publication  without  re¬ 
striction. 

The  Defense  Acquisition  University  does  not  accept  copy¬ 
righted  material  for  publication  in  Defense  AT&L.  Articles  will 
be  considered  only  if  they  are  unrestricted.  This  is  in  keep¬ 
ing  with  the  University's  policy  that  our  publications  be  fully 
accessible  to  the  public  without  restriction.  All  articles  are 
in  the  public  domain  and  posted  to  the  University's  website, 
www.dau.mil. 


http:  / /www.dau.mil /pubscats  /pages  /defenseatl.aspx 


DEFENSE  AT&L 


Learn,  Perform,  Succeed, 


